Transformer Library Documentation

Getting Started

• Installation
  • Requirements
  • Install from source
  • Dependencies
• Quick Start
  • Basic Usage
  • Training Loop Example
  • Using HuggingFace Generation
  • Returning Intermediate States

Guide

• Transformer: A PyTorch SOTA Transformer Implementation
  • Introduction
  • Installation
  • Quick Start
• Configuration
  • Default Configuration (SOTA Design)
  • Module Overview
    • 1. Attention Modules
    • 2. Feed‑Forward Modules
    • 3. Positional Embedding
    • 4. Transformer Components
  • Detailed Module Usage
    • Using MHA or GQA
    • Using SwiGLU
    • Using RoPE independently
    • Using TransformerBlock
  • Training a Model
  • Text Generation
  • Saving and Loading
  • Using Flash Attention
  • Accessing Intermediate States
    • The hidden_states[1] is a list of Dictionaries for each layer containing the following dictionary structure for each layer:
  • Custom Attention Mask
  • Extending the Library
    • Adding a new attention mechanism
    • Adding a new feed‑forward variant
    • Adding a new positional encoding
  • FAQ / Troubleshooting
  • License
  • Citation

API Reference

• API Reference
  • Configuration
  • Attention Modules
  • Positional Embeddings
  • Feed-Forward Modules
  • Transformer Model
  • LoRA (Parameter-Efficient Fine-Tuning)
  • Utilities

Usage Examples

• Usage Examples
  • Basic Usage
    • Text Processing
    • Image Processing with Vision Transformer (ViT)
  • Visualization
  • Training a Simple Model
  • Text Generation with HuggingFace GenerationMixin
  • Using Flash Attention
  • Custom Attention Mask
  • Encoder-Decoder Model (Seq2Seq)
    • Incremental Decoding with KV-Cache
  • Applying LoRA Adapters
    • LoRA on Multiple Module Types

Project Info

• Contributing
  • Development Setup
  • Code Style
  • Testing
  • Documentation
  • Pull Request Process
  • Code of Conduct

Overview

A Polished PyTorch implementation of the current State-Of-The-Art(SOTA) Transformer, designed to be a Baseline for Research and Engineering.

Features:

• Fully configurable architecture (layers, heads, dimensions, etc.)

• HuggingFace-compatible API (PreTrainedModel, GenerationMixin)

• Multi-Head Attention (MHA), Grouped-Query Attention (GQA) and Cross-Attention

• Rotary Position Embeddings (RoPE), PartialRoPE, and ALiBi

• SwiGLU and MLP feed-forward networks

• Encoder-Decoder architecture support with cross-attention

• Vision Transformer (ViT) support for image processing

• LoRA integration for parameter-efficient fine-tuning

• KV-Cache support for fast incremental decoding

• Optional weight tying, QK normalization, and bias control

• Flash Attention support for accelerated training and inference

Quick Example

Text Processing

import torch
from transformer import Transformer, TransformerConfig

config = TransformerConfig(vocab_size=32000, n_layers=12, n_heads=16, d_model=1024)
model = Transformer(config)

input_ids = torch.randint(0, 32000, (2, 512))
outputs = model(input_ids=input_ids)
logits = outputs.logits  # shape: (2, 512, 32000) [batch_size, seq_len, vocab_size]

Image Processing with Vision Transformer (ViT)

import torch
from transformer import Transformer, TransformerConfig

# Configure ViT with patch_size and img_size
config = TransformerConfig(
    vocab_size=1000,  # Output vocabulary for classification
    n_layers=12,
    n_heads=16,
    d_model=1024,
    patch_size=16,     # Patch size for image tokenization
    img_size=224,      # Input image size (can be int or tuple)
    in_channels=3,     # Number of input image channels (RGB)
    max_seq_len=512    # Must accommodate num_patches + 1 (cls token)
)
model = Transformer(config)

# Process images: shape (batch_size, channels, height, width)
images = torch.randn(2, 3, 224, 224)
outputs = model(images=images)
logits = outputs.logits  # shape: (2, 197, 1000) [batch_size, num_patches+1, vocab_size]

Indices and tables

• Index

• Module Index

• Search Page