 Command

Sam Foreman's personal site. Vim-style keybinds for navigation; theme + font pickers below.

Theme
 Font Body Code
Reader
Keybinds
Navigation
j / ↓ Next item k / ↑ Previous item g First item in region G Last item in region zz Center focused item h / l Sidebar / main content ] / [ Next/previous heading } / { Next/previous block d / u Half-page down/up
Layout
<zh> Toggle sidebar <zr> Toggle reader view <zj> / <zk> Focus main / actions ⇧C / ⇧E  ·  <zM> / <zR> Collapse / expand all sections
Dialogs
⌃P / : Command palette ⌃X Theme picker / Search ? Show keybinds ⌃N / ⌃P Next/prev search result Esc Close dialog / exit reader
History
n Next document b Previous document ⌃O History back ⌃I History forward
Sections
a about p posts t talks m more s style
 Search
about/more: 🪪 More about: Sam Foreman ideas: 💡 Ideas more: ➕ More now: Now posts: 📬 Posts talks: 🎙️ Talks projects: 📚 Projects webtui: Style posts/ai-for-physics: ⚛️ AI for Physics posts/2025: 📆 2025 posts/auroragpt: 🤖 AuroraGPT posts/dope-slides: 💅 How to Make Dope Slides posts/ezpz-at-alcf: 🍋 ezpz @ ALCF posts/jupyter: 📗 Jupyter posts/resume: 🧑🏻‍💻 Sam Foreman’s Résumé posts/svgbob: 🫥 svgbob posts/torchtune-aurora: 🪛 Torchtune on Aurora posts/ezpz-v1: 📝 ezpz-v1 posts/torchtune-patch-aurora: 🚑 Torchtune Patch on Aurora talks/auroragpt-siam25: AuroraGPT talks/aurora-gpt-fm-for-electric-grid/auroragpt-fm-for-electric-grid: AuroraGPT: Foundation Models for Science talks/ai-for-science-2024: Parallel Training Methods talks/hpc-user-forum/auroragpt: AuroraGPT talks/incite-hackathon-2025: ALCF Incite Hackathon 2025 talks/llms-at-scale: Training LLMs at Scale talks/llms-on-polaris: Training LLMs on Polaris talks/openskai25: Open SkAI2025 talks/alcf-hpc-workshop-2024/alcf-hpc-workshop-2024: Deep Learning and Foundation Models at Scale talks/demo-slides: AuroraGPT: Training Foundation Models on Supercomputers webtui/contributing/contributing: Contributing webtui/contributing/contributing: ## Local Development webtui/contributing/contributing: ## Issues webtui/contributing/contributing: ## Pull Requests webtui/contributing/style-guide: Style Guide webtui/contributing/style-guide: ## CSS Units webtui/contributing/style-guide: ## Selectors webtui/contributing/style-guide: ## Documentation webtui/components/accordion: Accordion webtui/components/badge: Badge webtui/components/checkbox: Checkbox webtui/components/button: Button webtui/components/input: Input webtui/components/dialog: Dialog webtui/components/popover: Popover webtui/components/pre: Pre webtui/components/progress: Progress webtui/components/radio: Radio webtui/components/separator: Separator webtui/components/range: Range webtui/components/spinner: Spinner webtui/components/textarea: Textarea webtui/components/switch: Switch webtui/components/tooltip: Popover webtui/components/table: Table webtui/components/typography: Typography webtui/components/view: View webtui/installation/astro: Astro webtui/installation/astro: ## Scoping webtui/installation/astro: ### Frontmatter Imports webtui/installation/astro: ### ‹style› tag webtui/installation/astro: ### Full Library Import webtui/installation/nextjs: Next.js webtui/start/ascii-boxes: ASCII Boxes webtui/installation/vite: Vite webtui/start/changelog: Changelog webtui/start/intro: Introduction webtui/start/intro: ## Features webtui/start/installation: Installation webtui/start/installation: ## Installation webtui/start/installation: ## Using CSS webtui/start/installation: ## Using ESM webtui/start/installation: ## Using a CDN webtui/start/installation: ## Full Library Import webtui/start/installation: ### CSS webtui/start/installation: ### ESM webtui/start/installation: ### CDN webtui/start/plugins: Plugins webtui/start/plugins: ## Official Plugins webtui/start/plugins: ### Themes webtui/start/plugins: ## Community Plugins webtui/start/theming: Theming webtui/start/theming: ## CSS Variables webtui/start/theming: ### Font Styles webtui/start/theming: ### Colors webtui/start/theming: ### Light & Dark webtui/start/theming: ## Theme Plugins webtui/start/theming: ### Using Multiple Theme Accents webtui/start/tuis-vs-guis: TUIs vs GUIs webtui/start/tuis-vs-guis: ## Monospace Fonts webtui/start/tuis-vs-guis: ## Character Cells webtui/plugins/plugin-dev: Developing Plugins webtui/plugins/plugin-dev: ### Style Layers webtui/plugins/theme-catppuccin: Catppuccin Theme webtui/plugins/plugin-nf: Nerd Font Plugin webtui/plugins/theme-everforest: Everforest Theme webtui/plugins/theme-custom: Custom Theme webtui/plugins/theme-vitesse: Vitesse Theme webtui/plugins/theme-nord: Nord Theme posts/ai-for-physics/diffusion: 🎲 MCMC + Diffusion Sampling webtui/plugins/theme-gruvbox: Gruvbox Theme posts/2025/06: 06 posts/auroragpt/determinstic-flash-attn/deterministic-flash-attn: 🎰 Deterministic flash-attn posts/ai-for-physics/l2hmc-qcd: 🎢 L2HMC for LQCD posts/auroragpt/aurora-gpt: 🏎️ Megatron-DeepSpeed on Intel XPU posts/auroragpt/mpi4py-reproducer: 🐛 mpi4py bug on Sunspot posts/auroragpt/flash-attn-sunspot: 📸 flash-attn on Sunspot posts/auroragpt/spike-skipper: 🏔️ Spike Skipper posts/auroragpt/startup-times: 🐢 Starting Up Distributed Training on Aurora posts/auroragpt/startup-times: ## Response posts/auroragpt/startup-times: ### Measuring / Calculating Startup Time posts/auroragpt/startup-times: ## Minimal Working Example posts/auroragpt/checkpoints: 💾 Converting Checkpoints posts/auroragpt/long-sequences: 🚂 Loooooooong Sequence Lengths talks/incite-hackathon-2025/auroragpt: LLMs on Aurora: Overview talks/auroragpt/alcf-hpc-workshop-2024/auroragpt-alcf-hands-on-hpc-workshop-2024: AuroraGPT: ANL's General Purpose Scientific LLM talks/incite-hackathon-2025/ezpz: LLMs on Aurora: Hands-On posts/jupyter/test: 🏁 l2hmc Example: 2D $U(1)$ posts/2026/01/07: 🎉 Happy New Year! posts/2026/01/10: 🍋 ezpz: distributed PyTorch across any hardware talks/openskai25/training: Scientific AI at Scale: Distributed Training posts/2026/02/28: ⏱️ Comparing Launchers on Aurora posts/2026/02/28: ## torchrun posts/2026/02/28: ## ezpz talks/openskai25/ai4science: Scientific AI at Scale: AuroraGPT posts/2026/05/01: Running 50k Python Processes on Aurora with ezpz yeet posts/2026/04/27: Pre-Training AuroraGPT with TorchTitan posts/2026/04/27: ## Two-Week Summary (Apr 12–27, 2026) posts/2026/04/27: ## Detailed Breakdown posts/2026/04/27: ### Week 1: Apr 12–18 — Benchmarking, LR Finder, XPU Fixes posts/2026/04/27: #### Benchmarking (Apr 12–15) posts/2026/04/27: #### LR Finder (Apr 12–14) posts/2026/04/27: #### Scaling Study (Apr 12) posts/2026/04/27: #### Upstream Syncs (Apr 12–18, syncs 6–14) posts/2026/04/27: #### XPU Bug Fixes (Apr 18) posts/2026/04/27: #### RL Experiment (Apr 18) posts/2026/04/27: ### Week 1.5: Apr 18–25 — Production Readiness posts/2026/04/27: #### Torch 2.12 Benchmarks (Apr 18) posts/2026/04/27: #### LR Finder Extensions (Apr 20–21) posts/2026/04/27: #### XPU Fixes (Apr 23) posts/2026/04/27: #### Torch 2.13 Environment (Apr 25) posts/2026/04/27: #### 2B Scaling Study on Torch 2.13 (Apr 25) posts/2026/04/27: #### Production Training (Apr 25) posts/2026/04/27: ### Week 2: Apr 26–27 — Optimizer Competition posts/2026/04/27: #### RL Multi-Task Refactor (Apr 26) posts/2026/04/27: #### Docs Reorganization (Apr 26) posts/2026/04/27: #### Generic HF Dataset Streaming (Apr 26) posts/2026/04/27: #### New Optimizers (Apr 26) posts/2026/04/27: #### Architecture Tweaks (Apr 26–27) posts/2026/04/27: ## Competition Results posts/2026/04/27: ### Round 1–3: Speedrun — 2N, GBS=48, 1000 steps posts/2026/04/27: ### 10B Full Training — 8N, GBS=384, ~3,178 steps posts/2026/04/27: ### Round 4: Reproducible Speedrun — 2N, GAS=8, GBS=384, 1000 steps posts/2026/04/27: ## Key Discoveries posts/2026/04/27: ## Infrastructure Built posts/2026/04/27: ## High-Level posts/2026/04/27: ## Detailed Breakdown posts/2026/04/27: ### Week 1: Apr 12–18 — Benchmarking, LR Finder, XPU Fixes posts/2026/04/27: #### Benchmarking (Apr 12–15) posts/2026/04/27: #### LR Finder (Apr 12–14) posts/2026/04/27: #### Scaling Study (Apr 12) posts/2026/04/27: #### Upstream Syncs (Apr 12–18, syncs 6–14) posts/2026/04/27: #### XPU Bug Fixes (Apr 18) posts/2026/04/27: #### RL Experiment (Apr 18) posts/2026/04/27: ### Week 1.5: Apr 18–25 — Production Readiness posts/2026/04/27: #### Torch 2.12 Benchmarks (Apr 18) posts/2026/04/27: #### LR Finder Extensions (Apr 20–21) posts/2026/04/27: #### XPU Fixes (Apr 23) posts/2026/04/27: #### Torch 2.13 Environment (Apr 25) posts/2026/04/27: #### 2B Scaling Study on Torch 2.13 (Apr 25) posts/2026/04/27: #### Production Training (Apr 25) posts/2026/04/27: ### Week 2: Apr 26–27 — Optimizer Competition posts/2026/04/27: #### RL Multi-Task Refactor (Apr 26) posts/2026/04/27: #### Docs Reorganization (Apr 26) posts/2026/04/27: #### Generic HF Dataset Streaming (Apr 26) posts/2026/04/27: #### New Optimizers (Apr 26) posts/2026/04/27: #### Architecture Tweaks (Apr 26–27) posts/2026/04/27: ## Competition Results posts/2026/04/27: ### Round 1–3: 1000-step speedruns, 2 nodes, GBS=48 (17 configs) posts/2026/04/27: ### Round 4 (10B full training, 8 nodes, GBS=384, 5 configs) posts/2026/04/27: ### Round 5 (2 nodes, GAS=8, GBS=384, local dataset, 8 configs — in progress) posts/2026/04/27: ## Key Discoveries posts/2026/04/27: ## Infrastructure Built posts/2026/06/28: Migrating from Quarto to Astro: samforeman.me → samf.sh posts/2026/08/08: Pre-Training LLMs on a Supercomputer posts/2026/06/27: Local AI Apps on ALCF: Argo, Inference Endpoints, and One Gateway posts/2025/04/28: 🔥 Building PyTorch 2.6 from Source on Aurora posts/ai-for-physics/l2hmc-qcd/2du1: 🎢 l2hmc-qcd Example: 2D U(1) posts/2025/05/03: 🚧 Frameworks Issue with numpy \› 2 posts/2025/06/01: 📰 Nice Headings posts/2025/06/02: 🧜‍♀️ Mermaid posts/2025/06/14: 🏗️ Building PyTorch 2.8 from Source on Aurora posts/2025/10/06: 🎨 Mixing Between Distributions While Training posts/2025/09/17: 📊 pbs-tui: TUI for PBS Job Scheduler Monitoring posts/2025/11/12: 🧊 Cooling Down Checkpoints: Best Practices for Model Evaluation posts/2023/12/05: 🔳 l2hmc-qcd Example: 4D SU(3) posts/2025/09/12: 🍹 BlendCorpus + TorchTitan @ ALCF talks/2025/12/16: AuroraGPT: Training Foundation Models on Supercomputers talks/2025/10/08: AERIS: Argonne's Earth Systems Model talks/2025/09/24: Training Foundation Models on Supercomputers talks/2025/10/15: Training Foundation Models on Supercomputers talks/2025/10/24: Training Foundation Models on Supercomputers talks/2026/07/14: Pre-Training AuroraGPT at Scale on Aurora talks/2026/06/03: Production Pre-Training at Scale: The Good, the Bad, and the Restarts talks/2026/08/03: Pre-Training LLMs on a Supercomputer posts/drafts/2025/09/22: 📝 2025 Annual Report
 Theme Current: Light j/k or ↑/↓ + Enter

📝 2025 Annual Report

Draft annual report covering scientific accomplishments, publications, presentations, and goals at ALCF.

Goals for Next Year (2026)

  • Build out generic training services for science teams
  • Continue to push on resilient / fault-tolerant training techniques

Goals from Last Year (2024)

  • Continue to contribute to division(/lab)-wide efforts
  • Continue to work with application teams to efficiently scale on ALCF systems
  • [WIP] Publish retrospective on initial pre-training of AuroraGPT

Contributions to ALCF

  • AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictions

  • MProt-DPO: Breaking the ExaFLOPS Barrier for Multimodal Protein Design with DPO

  • AuroraGPT

    • Co-lead Models and Training team with Venkat Vishwanath
    • Ongoing writeup of pre-training efforts
    • Successfully pre-trained:
      • AuroraGPT-7B on 2T tokens
      • AuroraGPT-2B on 4T tokens (ongoing)
  • Catalyst for:

    • Arvind Ramanthan’s INCITE Project (FoundEpidem)
    • Zheng Zhang’s ALCC Project
    • Rao Kotamarthi’s ALCC Project
  • Member of Software Committee

  • Intro to HPC Undergraduate Bootcamp:

Publications

  1. AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictions (Hatanpää et al. (2025))1
  2. Aurora: Architecting Argonne’s First Exascale Supercomputer for Accelerated Scientific Discovery (Allen et al. (2025))
  3. HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights (Gokdemir et al. (2025))
  4. Automated Tuning for HMC Mass Ratios (Torsiello et al. (2025))
  5. MOFA: Discovering Materials for Carbon Capture with a GenAI and Simulation-Based Workflow (Yan et al. (2025))
  6. MProt-DPO: Breaking the ExaFLOPS Barrier for Multimodal Protein Design with DPO (Dharuman et al. (2024))2

Presentations

Posts

Organizational Efforts

Mentoring

  • Khalid Hossain: Supported Khalid’s successful transition from postdoc to staff
  • Joseph Frimpong: Postdoc in Center for Nanoscale Materials
  • Hung Nguyen: Graduate student @ UIUC

Scientific / Technical Accomplishments

References

Allen, Benjamin S., James Anchell, Victor Anisimov, et al. 2025. Aurora: Architecting Argonne’s First Exascale Supercomputer for Accelerated Scientific Discovery. https://arxiv.org/abs/2509.08207.

Dharuman, Gautham, Kyle Hippe, Alexander Brace, et al. 2024. “MProt-DPO: Breaking the ExaFLOPS Barrier for Multimodal Protein Design Workflows with Direct Preference Optimization.” Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (Atlanta, GA, USA), SC ’24. https://doi.org/10.1109/SC41406.2024.00013.

Gokdemir, Ozan, Carlo Siebenschuh, Alexander Brace, et al. 2025. HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights. https://arxiv.org/abs/2505.04846.

Hatanpää, Väinö, Eugene Ku, Jason Stock, et al. 2025. AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictions. https://arxiv.org/abs/2509.13523.

Torsiello, J., G. T. Fleming, S. Foreman, X.-Y. Jin, and J. C. Osborn. 2025. “Automated Tuning for HMC Mass Ratios.” In PoS. Argonne, ALCF; Argonne National Laboratory (ANL), Argonne, IL (United States); Temple U.; Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States). https://doi.org/10.22323/1.466.0052.

Yan, Xiaoli, Nathaniel Hudson, Hyun Park, et al. 2025. MOFA: Discovering Materials for Carbon Capture with a GenAI- and Simulation-Based Workflow. https://arxiv.org/abs/2501.10651.

Footnotes

  1. 2025 ACM Gordon Bell Prize for Climate Modeling Finalist

  2. 🌟 2024 ACM Gordon Bell Finalist

 samf.sh / posts / drafts / 2025 / 09 / 22 · Top 1:1