Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware

Pretraining a competitive ~206M dual-stream Diffusion Transformer from scratch on a single consumer GPU (NVIDIA RTX 4090) in ~160 hours for under $55.

Víctor Rodrigo González1, Pablo M. Olmos1, Ignacio Peis2, Alejandro Lancho1
1Department of Signal Theory and Communications, Universidad Carlos III de Madrid (UC3M), Spain • 2Technical University of Denmark (DTU), Denmark
206M Dual-Stream DiT 1× RTX 4090 ($53.20) SPRINT 75% Sparsification Continuous Aspect RoPE Decoupled Pixel Adaptation (DiP) Optimal Transport Flow Matching

Overview: Pareto Frontier & Visual Synthesis

Frontier text-to-image foundation models require multi-billion parameter DiTs and industrial clusters. Asuka-FM co-designs model architecture, attention sparsity, and memory pipelines to push the extreme low-compute Pareto efficiency frontier on consumer workstations.

Compute Pareto Frontier: Denoiser Training FLOPs vs DPG-Bench
(a) Denoiser training compute (ZFLOPs, log scale) vs. compositional alignment (DPG-Bench).
Model Params Hardware Budget GenEval ↑ DPG ↑
Industrial Foundation Models
SD3 Medium 2.0B Massive Cluster 0.620 84.08
Accessible Academic Baselines
MiniT2I-L/16 0.91B 8× H100 (27 Days) 0.883† 84.80
Asuka-FM (Consumer GPU, Open Data, No Prompt FT)
Asuka-512 (Ours) 0.21B 1× RTX 4090 (81h) 0.609 78.51
(c) Representative benchmark summary across foundation models and academic baselines (†denotes fine-tuning on BLIP3o-60K).

Zero-Shot Synthesis Showcase

Asuka-FM High-Quality Zero-Shot Samples

Anime Domain: Zero-shot synthesis from our 206M dual-stream backbone pretrained on 337K Danbooru illustrations. Demonstrates sharp linework, anatomical consistency, and multi-concept binding without human aesthetic post-filtering.

1. Multi-Stage Research & Training Pipeline

Click any completed stage below to inspect exact epoch schedules, token drop ratios, throughput, and compute budgets measured on a single consumer NVIDIA RTX 4090 (24 GB, $0.33/hr):

1. Latent Pretraining (Completed)
→
2. Pixel Adaptation (Completed)
Native $v$-Prediction No Clipping • GenEval 0.568 / DPG 75.31
100K Synth → 396K Real SFT Up to 3.34× Faster at $N=1$ Step
→
3. Post-Training (Future Work)
Zero-Decode Flow-GRPO Online RL without VAE Rollout Wall
Few-Step Distillation $N \le 4$ Steps ($1.44\times$–$3.34\times$ Speedup)

Dual-Stream SPRINT Transformer Architecture (206.12M Pretraining Params)

Patchify (p=2) Aspect RoPE Encoder Blocks (2× Dense Layers) AdaLN-Free MM-Attn Sandwich RMSNorm Full Token Set N LFM2.5-230M + Adapter [z_time || c_text] -75% Deep Middle Path 9× Sparse (25% Tokens) 78.4% of Backbone Weights Scatter + [M] Dense Shallow Skip Connection f_t Residual Fusion W_fuse[f_t || g_pad] + 2× Decoder Blocks Unpatchify or PDH (RGB)
1. Image Input 512 × 512 × 3 Shifted Static Crop
→
2. FLUX.2 VAE 64 × 64 × 32 f=8 Latent Space
→
3. Patchify (p=2) 32 × 32 = 1024 Effective P=16 Tokens
→
4. SPRINT Subsample 256 Active Tokens 75% Drop (2×2 Group)
→
5. Mask Restoration 1024 Fused Tokens Dense Skip + [M] Tokens

1.1 SPRINT Compute & Memory Frontier Explorer

& RTX 4090 Telemetry

Adjust target resolution, structured $2 \times 2$ drop ratio ($r$), and micro-batch size to inspect visual sequence length $N$, FLOP reduction against a homogeneous 13-layer DiT, and 1MP hardware limits:

Total Sequence ($N$) 1,024 tokens 32 × 32 Patch Grid
Active Middle Tokens 256 tokens 9 Sparse Blocks (25%)
Total DiT FLOP Savings 52.6% (2.11×) Self-Attn FLOPs: -64.9%
Est. Peak VRAM (4090) 21.4 GB Selective SwiGLU Ckpt
Structured 2×2 Token Map
Active Dropped
⚠ Native 1MP Bottleneck Profiling ($1024^2, N = 4096$)
1. Online VAE Wall (72.6%): Caching 337K 1MP latents requires >1.2 TB NVMe. On-the-fly encoding consumes 72.6% of step time on A40 (3.66% sys MFU) and 62.0% on RTX 4090 ($B=8$).
2. Receptive-Field Starvation: Two dense layers ($L_{\text{enc}}=2$) span 12.5% of the canvas at $256^2$ but only 3.1% at $1024^2$. Dropping 75% (3,072 tokens) creates a 32px stride that fractures multi-character anatomy until dense Phase 3B recovery (53h).
3. Mitigation Recipe: 1MP sparsification requires a density curriculum (starting at $r=0.50$ or $L_{\text{enc}} \ge 4$), native $\ge 1024\text{px}$ filtering, or bypassing the VAE via latent-to-pixel adaptation.

1.2 SPRINT-Accelerated Latent-to-Pixel Adaptation (DiP)

Completed • 57.65h ($19.04)

Transferring the converged 512px foundation model directly to raw RGB pixels using a 10.09M Patch Detailer Head (PDH) under native velocity regression ($v$-prediction), permanently bypassing VAE decoding during few-step inference ($N \le 8$) and RL rollouts:

Noisy RGB Patch p_i (16×16×3) Shallow Path 2× Dense Blocks -75% Deep Middle Path 9× Sparse Blocks (25% Token Budget) Restore [M] + 2× Dec + Skip Dense Shallow Skip Path f_t Patch Detailer Head (PDH) 4-Stage Conv U-Net (+10.09M Params) Conditioned on Token s_i (D=768) Direct v-Prediction → RGB Direct Noisy RGB Patch Injection p_i (Upsample + 3×3 Conv)
Native $v$-Pred Beats $x_0$-Pred: Direct velocity regression avoids $t \to 0$ `bfloat16` singularities and dynamic clipping blowout, achieving 0.568 GenEval (vs. 0.545 for $x_0$-pred) and 0.420 color attribution (vs. 0.348).
3-Stage Adaptation Curriculum: PDH warmup (2.25h) and SPRINT joint transfer (21.4h) use 100K synthetic anchors, then Stage 3 (34.0h) refines densely on 396K real $\ge 1024\text{px}$ images (>95% latent score recovery).
Amortization Asymmetry: SPRINT speeds Stage 2A training by 1.30× (57.7 vs. 44.4 img/s). At inference, pixel mode is 3.34× faster at $N=1$ and 1.44× at $N=4$, whereas latent+VAE wins for multi-step ODE ($N \ge 13$, $N^* \approx 12.25$).

2. Theoretical Foundations: Continuous RoPE & Amortization Asymmetry

Two core analytical results govern Asuka-FM's spatial generalization and latent-to-pixel deployment regimes.

2.1 Continuous RoPE Phase Collapse & Calibration

Mapping a latent patch grid ($N_H \times N_W$) to area-preserving continuous coordinates $(y_{\text{pos}}, x_{\text{pos}}) \in [-r_H, r_H] \times [-r_W, r_W]$ with $r_H = \sqrt{N_H/N_W}$ and $r_W = \sqrt{N_W/N_H}$ enables arbitrary aspect ratios without dynamic padding or torch.compile graph breaks. However, naive continuous coordinates break standard RoPE frequencies:

Why Uncalibrated Continuous RoPE Collapses

$$\Delta y_{\text{norm}} = \frac{2r_H}{N_H} \ll 1 \;\Longrightarrow\; K(\mathbf{p}_i, \mathbf{p}_{i+1}) = 1.0 - \mathcal{O}(N_H^{-2})$$

Under standard geometric frequencies ($\omega_k = 10000^{-k/d_s} \le 1.0$), normalizing coordinates shrinks adjacent phase shifts below $7.2^\circ$. As $N_H$ grows, the cosine kernel $K(\mathbf{p}_i, \mathbf{p}_{i+1}) \to 1.0$, erasing local spatial discrimination and causing duplicated heads/torsos.

Parameter-Free Full-Spectrum Calibration

$$\omega_k = \exp\!\left(\ln(f_{\max}) - \frac{k}{d_s - 1}\ln\!\left(\frac{f_{\max}}{f_{\min}}\right)\right), \quad f_{\max} = 2\pi,\; f_{\min} = \frac{\pi}{4}$$

Bounding $f_{\max} = 2\pi$ enforces $45^\circ$ adjacent dispersion at $256^2$ without Nyquist aliasing, while $f_{\min} = \pi/4$ prevents half-cycle periodic resonance across $\Delta p_{\max} = 2.0$. Registered as a static buffer (zero learnable parameters or optimizer overhead).

Zero-Shot Virtual Camera Inversion: $$y'_{\text{pos}} = \frac{y_{\text{pos}}}{\text{zoom}} + (y_{\text{shift}} \cdot 2r_H), \qquad x'_{\text{pos}} = \frac{x_{\text{pos}}}{\text{zoom}} + (x_{\text{shift}} \cdot 2r_W)$$
Show Taylor Expansion Proof of Phase Collapse
For $d_s = d_h / 4$ rotation channels per axis, the normalized 2D axial RoPE kernel between patches $\mathbf{p}_i$ and $\mathbf{p}_j$ is: $$K(\mathbf{p}_i, \mathbf{p}_j) = \sum_{k=0}^{d_s-1} \frac{\cos\big((y_i - y_j)\omega_k\big) + \cos\big((x_i - x_j)\omega_k\big)}{2d_s}$$ For a diagonal step $\Delta x_{\text{norm}} = \Delta y_{\text{norm}} = 2r_H/N_H$ with $\omega_k \le 1.0$, adjacent phase shifts satisfy $\Delta\theta_k \le 2r_H/N_H \ll 1.0$. Expanding $\cos(\Delta\theta_k) = 1 - \frac{1}{2}(\Delta\theta_k)^2 + \mathcal{O}(N_H^{-4})$ and summing over $d_s$ channels yields: $$K(\mathbf{p}_i, \mathbf{p}_{i+1}) = 1 - \frac{1}{2d_s}\sum_{k=0}^{d_s-1} (\Delta\theta_k)^2 + \mathcal{O}(N_H^{-4}) = 1.0 - \mathcal{O}(N_H^{-2}) \xrightarrow{N_H \to \infty} 1.0.$$

2.2 Latent vs. Pixel Sampling: The Amortization Asymmetry

Removing the VAE decoder does not universally speed up inference. While latent diffusion pays a one-time $O(1)$ terminal VAE decode penalty ($t_{\text{fixed, VAE}}$), pixel diffusion must evaluate the 10.09M convolutional PDH head recurrently at every ODE step (adding $\Delta t_{\text{head}}$ per step, twice per step under CFG):

$$T_{\text{latent}}(N) = N \cdot t_{\text{step, latent}} + t_{\text{fixed, VAE}}, \qquad T_{\text{pixel}}(N) = N \cdot (t_{\text{step, latent}} + \Delta t_{\text{head}}) + t_{\text{fixed, pixel}}$$ $$N^* = \frac{t_{\text{fixed, VAE}} - t_{\text{fixed, pixel}}}{t_{\text{step, pixel}} - t_{\text{step, latent}}} \;\xrightarrow{\text{1MP, } B=16}\; N^* = \frac{4.82\,\text{s} - 0.09\,\text{s}}{1.773\,\text{s} - 1.387\,\text{s}} \approx \mathbf{12.25\text{ steps}}$$
NVIDIA A40 • Batch B=16 • TorchInductor
Latent DiT + VAE Decode 10.37 s 4 × 1.387s + 4.82s VAE
Pixel DiP (Zero VAE) 7.18 s 4 × 1.773s + 0.09s fixed
Operational Winner Pixel 1.44× Faster Few-step regime (N < 12.25)
Latent + VAE
Pixel DiP

Key Takeaway: Pixel diffusion ($P=16$) is decisively faster in few-step / distilled regimes ($3.34\times$ at $N=1$, $1.44\times$ at $N=4$ at 1MP), whereas latent sampling wins for multi-step ODE trajectories ($N \ge 13$ at 1MP, $N \ge 6$ at $512^2$, where pixel is $9\%$–$31\%$ slower)—unless patch size is widened to $P=32$ ($1.78\times$ sampling speedup at $512^2$).

3. Empirical Validation & Benchmark Analysis

Quantitative evaluations, consumer vs datacenter hardware profiling, pretraining timings, and latent-to-pixel transfer benchmarks across asuka-fm (625K open corpus & 337K Danbooru) and toy-diffusion:

Open-Domain Compositional Benchmarks & Compute Cost

Evaluated strictly as an open-domain foundation model (100-step Euler ODE, $w_{\text{cfg}} = 8.0$ at $256^2$ and $6.0$ at $512^2$) with no benchmark-specific fine-tuning. Models marked † were fine-tuned on BLIP3o-60K, which directly aligns prompts to GenEval syntax.

Model Params Hardware Budget (Est. Cost) Training Data GenEval ↑ DPG ↑
Industrial Foundation Models (Multi-Billion Parameters, Datacenter Compute)
SDXL Base 2.6B Hundreds of A100s Millions (Web) 0.406* 72.40*
SD3 Medium 2.0B Massive Cluster (>$70k) Millions (Web) 0.620* 84.08*
FLUX.1 [Dev] 12.0B Massive Cluster (>$175k) Web-Scale 0.660* 83.84*
Janus-Pro-7B 7.0B Large Cluster Multimodal Web 0.800* 84.19*
i1 (Open Foundation) 3.0B TPU v5p-128 (90.6k h, ≈$380k) Curated Open 0.840 86.73
Accessible Academic Baselines (Sub-2B Parameters, Finetuned on BLIP3o-60K†)
PixNerd-XXL 1.2B Multi-GPU Cluster Pretrain + BLIP3o† 0.730* 80.90*
PixelGen-XXL 1.1B 8× H800 (1,152 h, ≈$2,880) 36M + BLIP3o† 0.790* 79.36
DeCo-XXL 1.1B Multi-GPU Cluster Pretrain + BLIP3o† 0.860* 81.40*
Looped-DiT B/16 0.26B 8× H100 (576 h, ≈$2,016) CC12M + BLIP3o† 0.874* 87.00*
MiniT2I-L/16 0.91B 8× H100 (27 Days) CC12M + BLIP3o† 0.883* 84.80*
Asuka-FM (Ours: 1× Consumer RTX 4090, 625K Open Curated Corpus, No BLIP3o Finetuning)
Asuka-256 (Dense Final) 0.21B 1× RTX 4090 (68.0h, $22.44) 625K Open 0.588 78.96
Asuka-512 (75% Sparse, Ph. 2A) 0.21B 1× RTX 4090 (40.0h, $13.20) 625K Open 0.480 68.40
Asuka-512 (Dense FT 20k Steps) 0.21B 1× RTX 4090 (60.0h) 625K Open 0.575 79.49
Asuka-512 (Dense Final, Ep 139) 0.21B 1× RTX 4090 (81.0h, $26.73) 625K Open 0.609 78.51

*Prior literature numbers. Total latent pretraining compute is 149.0h ($49.17) DiT-only (0.0657 ZFLOPs), or 161.2h ($53.20) including one-time offline VAE encoding.

Order-of-Magnitude Cost Reduction: Asuka-FM trains from scratch for $49.17–$53.20 on one consumer GPU vs. $2,016–$2,880 for 8× H100/H800 baselines and $70k–$380k for industrial clusters, while outperforming SDXL on both GenEval (0.609 vs. 0.406) and DPG-Bench (78.51 vs. 72.40).
Sparse Layout vs. Dense Attribute Binding: In Phase 2A (75% token drop), Single-Object remains strong (0.963) while Color-Attribution collapses (0.163). Switching to full-sequence dense refinement (Phase 2B) surges DPG-Bench to 79.49 at 20k steps and lifts Color-Attr to 0.450 and Two-Object to 0.811 by Epoch 139.

Multi-Stage Pretraining Progression (625K Open Corpus)

Measured on a single consumer NVIDIA RTX 4090 (24 GB, $0.33/hr) with bfloat16, FlashAttention-2, static torch.compile, 8-bit AdamW, and effective batch size 256 (Tables 2 & 8; sequence lengths include 227 text + 1 time token):

Stage / Phase Res. Drop ($r$) Epochs Samples Throughput Token Rate MFU (Theo / Act) EFLOPs Time Cost
Stage 1: Base Semantic Pretraining (256 × 256, Latent Patch Grid 16 × 16 = 256 Image Tokens)
Offline VAE Encoding‡ $256^2$ – – 0.625M 45.2 img/s – – – 3.8 h $1.27
1A: SPRINT Sparse $256^2$ 0.75 75 46.88M 282.6 img/s 99.2k tok/s 74.4% / 86.8% 23.74 47.0 h $15.51
1B: Dense Refine (+1C WSD) $256^2$ 0.00 20 12.50M 172.5 img/s 83.5k tok/s 62.9% / 73.4% 8.73 21.0 h $6.93
Stage 2: High-Resolution Adaptation (512 × 512, Latent Patch Grid 32 × 32 = 1,024 Image Tokens)
Offline VAE Encoding‡ $512^2$ – – 0.625M 20.6 img/s – – – 8.4 h $2.78
2A: High-Res Sparse $512^2$ 0.75 25 15.63M 108.5 img/s 78.1k tok/s 58.5% / 68.3% 16.24 40.0 h $13.20
2B: Dense Refine (+2C WSD) $512^2$ 0.00 15 9.38M 64.8 img/s 81.1k tok/s 61.0% / 71.2% 16.94 41.0 h $13.53
Cumulative Totals
DiT Training Total (Stages 1 + 2) – – 135 84.38M – – 64.8% / 75.6% 65.65 149.0 h $49.17
Grand Total (Inc. Offline VAE) – – – – – – – 65.65 161.2 h $53.20

Consumer RTX 4090 vs. Datacenter NVIDIA A40 & Online VAE Wall

Despite half the VRAM (24 GB vs. 48 GB), the RTX 4090 executes compiled latent steps 1.97× faster than the A40 because its 12× larger L2 cache (72 MB vs. 6 MB) keeps TorchInductor Triton kernels on-chip and avoids GDDR6 fallback (1,008 vs. 696 GB/s). Conversely, on-the-fly FLUX.2 VAE encoding at high resolutions starves the GPU:

Workload / Device Res. (BS) Peak VRAM Online VAE Text Enc. Backbone (F+B) Opt. & Clip Total Step Sys. MFU
NVIDIA A40 (In-RAM Latents) $256^2$ (BS 64) 13.25 GB – 99.98 ms 469.88 ms 36.33 ms 607.65 ms 31.71%
RTX 4090 (In-RAM Latents) $256^2$ (BS 64) 15.25 GB 0.28 ms 38.94 ms 247.91 ms 20.87 ms 308.27 ms 63.25%
NVIDIA A40 (Online VAE Wall) $512^2$ (BS 64) 35.20 GB 2.264 s (59.4%) 111.34 ms 1,396.95 ms 36.28 ms 3,808.42 ms 15.27%
NVIDIA A40 (1MP Online VAE) $1024^2$ (BS 16) – 2.548 s (72.6%) 31.84 ms 888.96 ms (25.3%) 38.49 ms 3,507.55 ms 3.66%
Curated Danbooru 337K Multi-Stage Timings (256² → 512² → 1024²)

On the 337K Danbooru testbed (effective batch size 256 for Stages 1–2, 128 for Stage 3; 152 text + 1 time = 153 tokens), Stages 1 + 2 ($256^2 \to 512^2$) complete in 110.0h ($36.30), and native $1024^2$ scaling adds 89.0h ($29.37) for 199.0h ($65.67) total:

Stage / Phase Res. Drop ($r$) Epochs Samples Throughput MFU (Theo / Act) EFLOPs Time Cost
Stage 1: Base Semantic Pretraining (256 × 256, 256 Image Tokens)
1A: SPRINT Sparse $256^2$ 0.75 100 33.70M 334.3 img/s 69.1% / 80.6% 13.42 28.0 h $9.24
1B: Dense Refine $256^2$ 0.00 50 16.85M 234.0 img/s 71.7% / 83.6% 9.94 20.0 h $6.60
1C: WSD Cooldown $256^2$ 0.00 25 8.43M 234.0 img/s 71.7% / 83.6% 4.97 10.0 h $3.30
Stage 2: High-Resolution Adaptation (512 × 512, 1,024 Image Tokens)
2A: High-Res Sparse $512^2$ 0.75 25 8.43M 70.9 img/s 34.3% / 40.0% 7.84 33.0 h $10.89
2B: Dense Finetune $512^2$ 0.00 10 3.37M 72.0 img/s 63.4% / 74.0% 5.72 13.0 h $4.29
2C: WSD Cooldown $512^2$ 0.00 5 1.69M 78.0 img/s 68.7% / 80.2% 2.86 6.0 h $1.98
Stage 3: Native Megapixel Scaling (1024 × 1024, 4,096 Image Tokens)
3A: SPRINT Sparse $1024^2$ 0.75 3 1.01M 7.80 img/s 12.4% / 14.5% 3.10 36.0 h $11.88
3B: Dense Finetune $1024^2$ 0.00 4 1.35M 7.07 img/s 22.5% / 26.2% 8.26 53.0 h $17.49
Cumulative Totals
Cumulative 512² Subtotal (1 + 2) – – 215 72.46M – 58.7% / 68.4% 44.77 110.0 h $36.30
Cumulative Grand Total (1 + 2 + 3) – – 222 74.81M – 40.6% / 47.4% 56.13 199.0 h $65.67

Latent-to-Pixel Adaptation at $512^2$

Adapting the converged Asuka-512 latent backbone to raw RGB pixels using the 10.09M Patch Detailer Head (PDH) (effective batch size 128 on RTX 4090). All benchmarks evaluated at $512^2$ (100-step Euler ODE, $w_{\text{cfg}}=6.0$); per-image sampling latency profiled on NVIDIA A40 at batch size 32:

Configuration / Stage Patch ($P$) Obj. Schedule & Compute (4090) Img/s (MFUact) Colors Color-Attr GenEval ↑ DPG ↑ Latency (20s / 30s)
Asuka-512 (Base Latent) 16 $v$-pred 139 Ep (81.0h, $26.73) 64.8 (71.2%) 0.817 0.450 0.609 78.51 0.250s / 0.374s
Stage 1–2: Synthetic Anchor Transfer (100K Asuka-512 Synth Images; Phase 0 Gen on A40: 15.0h, $6.00)
Stage 1: PDH Warmup (Frozen DiT) 16 $v$-pred 5 Ep, $r=0.75$ (2.25h, $0.74) 61.8 (72.8%) – – – – –
Stage 2A: Joint SPRINT Sparse 16 $v$-pred 25 Ep, $r=0.75$ (12.0h, $3.98) 57.7 (67.8%) 0.662 0.215 0.465 69.63 0.329s / 0.493s
Stage 2B: Dense Refine (Synth) 16 $x_0$-pred 15 Ep, $r=0.00$ (9.4h) 44.4 (–) 0.628 0.233 0.477 71.49 0.329s / 0.493s
Stage 2B: Dense Refine (Synth) 16 $v$-pred 15 Ep, $r=0.00$ (9.4h, $3.10) 44.4 (69.8%) 0.726 0.273 0.505 72.35 0.329s / 0.493s
Stage 3: Real-Data SFT Refinement (396,635 Curated Real Images ≥1024px from i1; Synth Discarded)
Stage 3 SFT ($x_0$-pred Ablation) 16 $x_0$-pred 13 Ep, 70k steps (34.0h) 42.1 (–) 0.707 0.348 0.545 76.32 0.329s / 0.493s
Stage 3 SFT ($v$-pred, Final Pixel DiT) 16 $v$-pred 13 Ep, 70k (34.0h, $11.22) 42.1 (66.2%) 0.750 0.420 0.568 75.31 0.329s / 0.493s
Progressive Token Compression (P = 16 → 32; 4× Sequence Reduction to N = 256 Tokens)
ps32-adapt16 (Large Patch Engine) 32 $v$-pred +25 Ep (72.0h, $23.76) 38.3 (63.3%) 0.739 0.250 0.504 69.87 0.187s / 0.277s
>95% Anchor Recovery for $19.04: Stages 1–3 execute 15.15 EFLOPs in 57.65h on a single RTX 4090 (72.65h / $25.04 including Phase 0 A40 synth generation), recovering 0.568 GenEval / 75.31 DPG with zero VAE decoding.
Why Native $v$-Pred Wins: Converting $\hat{x}_0$ via $v_t = (x_t - \hat{x}_0)/t$ hits a $t \to 0$ bfloat16 singularity requiring dynamic percentile clipping that desaturates colors. Native $v$-pred avoids clipping entirely and improves GenEval (0.568 vs. 0.545) and Color-Attr (0.420 vs. 0.348).
$P = 32$ Speed-Quality Trade-Off: Widening patches to $P=32$ cuts tokens $4\times$ and speeds 30-step sampling by 1.78× (0.277s vs. 0.493s/img on A40), surpassing latent 30-step sampling (0.374s) at the cost of lower compositional binding (0.504 GenEval).

Controlled toy-diffusion Ablations (87K Portraits @ $256^2, P=16$)

Prior to foundation pretraining, architectural and conditioning choices were isolated on 87,000 facial portrait crops at $256 \times 256$:

Backbone / Architecture Variant Params BS Epochs FID ↓ Key Finding
Part I: Latent Architecture & Conditioning Ablations
Single-Stream DiT ($D=768, L=14$) 218M 256 600 3.89 Modality interference; lower MFU (49.8% vs. 70.4%)
Efficient U-Net Baseline 185M 256 400 3.45 Convolutional baseline
Dual-Stream DiT + MLP Text Adapter 200M 256 600 3.42 Static projection trails 2-layer bidirectional adapter (3.12)
Dual-Stream DiT + LLM (Frozen LFM2.5) 206M 512 1,000 3.12 AdaLN-free + prepended time token cuts params 28%
Dual-Stream Baseline (Constant LR / WSD) 206M 512 600 3.06 WSD linear decay drops FID from 3.27 → 3.06
Dual-Stream Baseline (Large Batch) 206M 1024 1,400 2.42 Best toy-scale latent convergence
Dual-Stream Wide ($D=1024, L=10$) 352M 512 750 2.85 Wider hidden dim with WSD cooldown (3.27 → 2.85)
SPRINT Dual-Stream ($r=0.75$, Sparse Only) 206M 256 700 5.56 52% faster iterations; plateaus without dense FT
SPRINT Dual-Stream ($r=0.75 \to$ Dense FT) 206M 256 1,000 3.96 Dense FT recovers fidelity at >50% cumulative FLOP savings
Part II: Early Latent-to-Pixel DiP Ablations ($x_0$-Prediction on Toy Scale; Superseded by $v$-Pred at $512^2$)
Stage 1: PDH Only (Frozen DiT Backbone) +10M 256 15 28.40 Warmup aligns convolutional head; faint patch seams remain
Stage 2: Joint FT ($P=16$, Hard Clamp) 216M 256 50 19.85 Hard $[-1, 1]$ clamp shifts RGB hue vectors
Stage 2: Joint FT ($P=16$, Dynamic Clip) 216M 256 20 / 50 / 85 18.20 / 16.00 / 14.39 Dynamic percentile clipping improves $x_0$-pred stability
Stage 2: Staged Patch Widening ($P=16 \to 32$) 216M 256 50 33.80 4× token cut without NaN divergence (direct $P=32$ diverges)

Toy-Diffusion Convergence Curve

Toy Diffusion Benchmarks

Evaluation of SPRINT token drop schedules ($r = 0.75 \to 0.00$) on FID. Sparse pretraining plateaus at FID 5.56, while terminal dense fine-tuning drops FID to 3.96 (vs. 3.89 Single-Stream and 3.06–3.12 Dual-Stream baseline) at >50% FLOP savings.

Resolution Scaling: 256px Base vs. 512px Adaptation

256px vs 512px Generations

Zero-shot qualitative comparisons across characters (Akane, Asuka Bodysuit, Hoshino Ai, Rae & Claire). 512px adaptation sharply resolves micro-textures, hair strands, and eye lineart.

4. Continuous Aspect RoPE & Virtual Camera Simulators

Interactive verification of Asuka-FM's parameter-free coordinate manifold: (4.1) zero-shot inference camera steering, (4.2) static square crop training to prevent torch.compile JIT graph breaks, and (4.3) full-spectrum frequency calibration to cure phase collapse.

4.1 Zero-Shot Virtual Camera Viewport Simulator (Inference)

Apply affine coordinate transformations directly to the continuous RoPE evaluation mesh at inference time without retraining or auxiliary camera tokens. Compare the transformed viewport (blue) against the canonical base frame (red dashed):

Canonical Base Viewport (x=0, y=0, z=1) Active Transformed Viewport
Query: [-1.00, 1.00] × [-1.00, 1.00]

4.2 Shifted Square Crop & Static Tensor Visualizer

Replicates Asuka-FM's shifted crop pipeline (following Patch Diffusion [12] & HDM [11]): isotropically scale the shorter edge, extract a fixed static crop window (strictly $256^2$ or $512^2$ during pretraining to prevent torch.compile JIT recompilations), and slice the continuous aspect-normalized RoPE map:

Resized Source Image 256 × 384
⟶
Extracted Crop Window (Training Pair) 256 × 256
Offset: (0, 0) $r_H$: 1.000 $r_W$: 1.000 RoPE Center $(x, y)$: (0.00, 0.00) Tensor Shape: Static Square (No JIT Break)

4.3 RoPE Phase Shift & Attention Kernel Inspector

Click any probe patch $(\star)$ on the $16 \times 16$ grid ($256^2$ canvas) to evaluate the normalized 2D axial rotary kernel $K(\mathbf{p}_i, \mathbf{p}_j) = \sum_{k=0}^{d_s-1} \frac{\cos((y_i - y_j)\omega_k) + \cos((x_i - x_j)\omega_k)}{2 d_s}$ (8 channels plotted per axis). Switch modes to diagnose phase collapse:

Spatial Attention Kernel $K(\mathbf{p}_i, \mathbf{p}_j)$ Probe: (8, 8)
-0.2 (Minimum)
1.0 (Maximum)
Adjacent Phase Shift $\Delta\theta_k$ Max: 45.0°
Discrete Broken Calibrated (Asuka-FM)
Mode C (Calibrated Continuous RoPE): Parameter-free frequencies ωk are scaled log-linearly from 2π down to π/4 (register_buffer). Adjacent patches separate by 45.0°, restoring sharp isotropic radial decay without optimizer overhead.

5. Empirical Discoveries & Hypothesis Defense

Empirical validation across open-domain and anime corpora covering zero-shot virtual camera dynamics, latent-to-pixel $v$-prediction vs. $x_0$-prediction, patch widening ($P=16 \to 32$), semantic attribute disentanglement, and ablation curtains:

5.1 Virtual Camera Flight, Manifold Support & Failure Modes

Continuous aspect-normalized RoPE enables smooth virtual camera panning and optical zoom under identical noise seeds. Switch between open-domain results, anime results, and zoom failure diagnostics:

Camera Viewport
Zoom: 1.00x | X: 0.00 | Y: 0.00 | Open-Domain 512px

Horizontal Panning: Shifting x_shift = +0.25 pans the viewport rightward, framing the dachshund toward the left canvas edge without altering knit cap details or snow texture.

Canonical Scale ($z = 1.0\times$) Sharp Structure
Unzoomed Cherry Blossom Scene (z=1.0)

At canonical unit scale ($z = 1.0\times$), the 206M backbone synthesizes coherent cobblestones, rigid wooden bench slats, and clean floral separation.

Continuous RoPE Zoom ($z > 1.0\times$) Physical Limits
Continuous RoPE Zoom Artifacts
Missing Asset:
Export zoomed bench grid to images/camera/failure_case_bench.png.

On complex organic compositions, magnifying via coordinate grid compression ($z > 1.0\times$) can induce: (i) painterly texture smearing on cobblestones, (ii) pink pigment bleeding, and (iii) bench-slat warping via rotary phase dampening ($\Delta\theta_k \propto 1/z$).

Manifold Support Discovery: Open-domain synthetic corpora (FLUX-Reason-6M, Midjourney-v6) are mainly centered $1024^2$ squares; shifted crops clip canonical focal points, causing warping on some prompts. Multi-aspect corpora (Danbooru, $1:2$ to $21:9$) span $[-r_H, r_H] \times [-r_W, r_W]$ and establish broader spatial invariance.

5.2 Latent-to-Pixel Adaptation: $v$-Pred vs. $x_0$-Pred & Patch Compression

Adapting the converged Asuka-512 latent backbone directly to RGB pixels via the 10.09M Patch Detailer Head (PDH) (57.65h on RTX 4090, $19.04) eliminates the online VAE bottleneck (59.4%–72.6% training wall; up to 3.34× faster 1-step inference). Below are the $512^2$ prediction objective, large-patch scaling ($P=16 \to 32$), and $256^2$ diagnostic ablations:

Stage 3 SFT ($x_0$-Prediction Ablation, $512^2$) GenEval 0.545
512px Pixel DiT x0-Prediction Samples
Add 512px $x_0$-Prediction Grid:
Place generated $512^2$ $x_0$-pred samples at images/comparisons/pixel/pixel_512_x0_pred.png.
GenEval: 0.545 Color-Attr: 0.348 Colors: 0.707 DPG: 76.32

Converting $\hat{x}_0$ into velocity targets $v_t = (x_t - \hat{x}_0)/t$ triggers $t \to 0$ singularities in bfloat16, forcing dynamic percentile clipping that flattens dynamic range and desaturates colors (Stage 2B Synth: 0.477 GenEval / 0.628 Colors).

Stage 3 SFT (Native $v$-Prediction, Final Pixel DiT) GenEval 0.568 (Zero Clip)
512px Final Pixel DiT Native v-Prediction Samples
Add 512px Native $v$-Prediction Grid:
Place generated $512^2$ $v$-pred samples at images/comparisons/pixel/pixel_512_v_pred.png.
GenEval: 0.568 Color-Attr: 0.420 Colors: 0.750 DPG: 75.31

Direct velocity regression maintains smooth probability paths with zero activation clipping, recovering >95% of the latent anchor (0.609 GenEval / 78.51 DPG) while boosting Two-Object (0.780 vs. 0.699) and Color Attribution (+0.072; Stage 2B Synth: 0.505 GenEval / 0.726 Colors).

$512^2$ Pixel DiT ($P = 16$, $N = 1,024$ Tokens) 0.568 GenEval | 0.493s (30s)
512px Pixel DiT P=16 Baseline
Add 512px $P=16$ Sample Grid:
Place generated $512^2$ $P=16$ samples at images/comparisons/pixel/pixel_512_p16.png.

Full Resolution Patch Grid ($P=16$): 1,024 tokens at $512^2$. Achieves 0.568 GenEval / 75.31 DPG (Color-Attr 0.420). A40 sampling latency (BS=32): 0.329s (20 steps) / 0.493s (30 steps).

$512^2$ Large Patch (`ps32-adapt16`, $P = 32$, $N = 256$) 1.78× Faster | 0.277s (30s)
512px Pixel DiT P=32 Large Patch Samples
Add 512px `ps32-adapt16` ($P=32$) Sample Grid:
Place generated $512^2$ $P=32$ samples at images/comparisons/pixel/pixel_512_p32.png.

4× Token Compression (+25 Ep, 72.0h, $23.76): Cuts sequence length to 256 tokens. Sampling drops to 0.187s (20s) / 0.277s (30s) on A40 (beating 30-step latent+VAE 0.374s), trading compositional binding (0.504 GenEval / 69.87 DPG) for ultra-fast throughput.

Toy-Scale $256^2$ Staged Widening ($P=16 \to 32$) Visuals & Divergence Matrix
Toy-Scale Configuration ($256^2$ Portraits) Tokens ($N$) Epochs FID ↓ Status
Latent DiT Baseline 256 ($p=2, f=8$) 1,000 2.42 – 3.06 Converged
Pixel DiP (Stage 2, $P=16$, Dyn. Clip) 256 ($16 \times 16$) 85 14.39 Sharp Lines
Pixel DiP (Direct $P=32$ from Scratch) 64 ($8 \times 8$) 1 ∞ Diverged (NaN)
Pixel `ps32-adapt16` (Staged $P=16 \to 32$) 64 ($8 \times 8$) 50 33.80 4× Token Cut
$256^2$ $P=16$ Baseline (256 Tokens) FID: 14.39
Toy 256px Patch Size 16 Grid
$256^2$ `ps32-adapt16` (64 Tokens) FID: 33.80
Toy 256px Patch Size 32 Grid
Why Toy-Scale Pixel FID Differs from Latent FID: Latent VAEs act as low-pass filters aligned with Inception-v3 statistics, whereas pixel models synthesize native high-frequency linework and subtle micro-grain that FID penalizes despite crisper visual edges. Staged adaptation ($P=16 \to 32$) stabilizes 4× compression where direct $P=32$ diverges immediately.

Controlled $256^2$ portrait ablations demonstrating (i) why freezing the DiT backbone during PDH warmup leaves faint grid seams until full joint fine-tuning, and (ii) why $x_0$-prediction suffers chromatic blowout without dynamic clipping—directly motivating native $v$-prediction at $512^2$:

Stage 1: PDH Head Only (Ep 5–15, FID 28.40) Frozen Backbone
Stage 1 PDH Head Only

Rapid convergence in <15 epochs. Colors and semantics stabilize quickly, but faint grid seams remain visible at $16 \times 16$ patch boundaries.

Stage 2: Joint Full FT (Ep 85, FID 14.39) End-to-End
Stage 2 Full Finetuned

Unfreezing the DiT backbone lets convolutional inductive bias propagate into attention layers, eliminating patch boundaries completely.

Clipping Ablation Output
Toy-Scale Objective $x_0$-prediction (superseded by $v$-pred at $512^2$)
Color Shift Diagnosis High-contrast blowout & color shift (FID > 50)
Observed Effect Unbounded x0 predictions extrapolate past [-1, 1], inducing severe chromatic shifts and contrast collapse.

5.3 Semantic Disentanglement & Compound Artist Style Steering

Verifying that our AdaLN-free Dual-Stream DiT decouples chromatic attributes from geometry and responds smoothly to compound artist tokens without explicit aesthetic SFT:

Pink Hair Variation Prompt: pink_hair, cat_ears, maid_outfit
Blue Hair Variation Prompt: blue_hair, cat_ears, maid_outfit

Under identical latent noise initializations $x_1 \sim \mathcal{N}(0, \mathbf{I})$, toggling only pink_hair → blue_hair alters hair pigment while leaving facial morphology, finger poses, ribbon folds, and sparkle lighting invariant down to the pixel level.

Style Variant A Albedo: Anime Coloring
Style Variant B Albedo: Compound Artist Tags

Injecting compound artist descriptors enriches volumetric rim lighting, textured black wing plumage, and gold filigree embroidery while preserving pose and facial identity.

5.4 Interactive Resolution, RoPE & SPRINT Ablation Curtain

Drag the slider across identical random seeds to inspect resolution scaling (256px → 512px), mid-flight RoPE calibration, and SPRINT sparse-to-dense recovery at 512px and 1MP:

After Adaptation
Before Adaptation
256px Base 512px Adapted

Canvas Scaling (256² → 512²): Adapting to 512×512 resolves intricate floral kimono patterns, umbrella ribs, and crisp iris highlights while preserving global pose and color harmony.

Citation

If you find Asuka-FM or our systems profiling useful for your research, please cite (arXiv identifier will be updated upon public posting):

@article{rodrigo2026asukafm,
  title   = {Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware},
  author  = {Rodrigo Gonz{\'a}lez, V{\'i}ctor and Olmos, Pablo M. and Peis, Ignacio and Lancho, Alejandro},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026},
  note    = {Forthcoming}
}