Transformer Library Documentation

Getting Started

• Installation
  • Requirements
  • Install from source
  • Dependencies
• Quick Start
  • Basic Usage
  • Training Loop Example
  • Using HuggingFace Generation
  • Returning Intermediate States

Guide

• Transformer: A PyTorch SOTA Transformer Implementation
  • Introduction
  • Installation
  • Quick Start
• Configuration
  • Default Configuration (SOTA Design)
  • Module Overview
    • 1. Attention Modules
    • 2. Feed‑Forward Modules
    • 3. Positional Embedding
    • 4. Transformer Components
  • Detailed Module Usage
    • Using MHA or GQA
    • Using SwiGLU
    • Using RoPE independently
    • Using TransformerBlock
  • Training a Model
  • Text Generation
  • Saving and Loading
  • Using Flash Attention
  • Accessing Intermediate States
    • The hidden_states[1] is a list of Dictionaries for each layer containing the following dictionary structure for each layer:
  • Custom Attention Mask
  • Extending the Library
    • Adding a new attention mechanism
    • Adding a new feed‑forward variant
    • Adding a new positional encoding
  • FAQ / Troubleshooting
  • License
  • Citation

API Reference

• API Reference
  • Configuration
  • Attention Modules
  • Positional Embeddings
  • Feed-Forward Modules
  • Transformer Model

Usage Examples

• Usage Examples
  • Basic Usage
  • Visualization
  • Training a Simple Model
  • Text Generation with HuggingFace GenerationMixin
  • Using Flash Attention
  • Custom Attention Mask

Project Info

• Contributing
  • Development Setup
  • Code Style
  • Testing
  • Documentation
  • Pull Request Process
  • Code of Conduct

Overview

A Polished PyTorch implementation of the current State-Of-The-Art(SOTA) Transformer, designed to be a Baseline for Research and Engineering.

Features:

• Fully configurable architecture (layers, heads, dimensions, etc.)

• HuggingFace-compatible API (PreTrainedModel, GenerationMixin)

• Multi-Head Attention (MHA), Grouped-Query Attention (GQA) and others

• Rotary Position Embeddings (RoPE) and SwiGLU feed-forward

• Optional weight tying, QK normalization, and bias control

Quick Example

import torch
from transformer import Transformer, TransformerConfig

config = TransformerConfig(vocab_size=32000, n_layers=12, n_heads=16, d_model=1024)
model = Transformer(config)

input_ids = torch.randint(0, 32000, (2, 512))
outputs = model(input_ids)
logits = outputs.logits  # shape: (2, 512, 32000) [batch_size, seq_len, vocab_size]

Indices and tables

• Index

• Module Index

• Search Page