Skip to content

Projects

Projects
Tooling2026★ 1

VRAMancer

Will this model fit, and how fast will it run?

A hardware-aware LLM/VLM feasibility estimator: parameter and quantization memory modelling, KV-cache and context scaling, and inference latency/throughput prediction across CPU and GPU backends.

//The problem

Working out whether a given model will actually run on given hardware — and at what speed — normally means downloading it and finding out.

//What it does

  • Models memory from parameter count and quantization scheme.
  • Accounts for KV-cache growth as context length scales.
  • Predicts latency and throughput across CPU/GPU backends including Ollama and Hugging Face.

//Measured

Backends
CPU, GPU, Ollama, Hugging Face

//Built with

  • Rust
  • Ollama
  • Hugging Face
  • Quantization

status: Active