FastInfer: Breaking the Low-to-Moderate Sparsity Barrier for Efficient LLM Inference

1. Clone this project.

git clone --recursive https://github.com/ParCIS/FastInfer.git

• Requirements:

• Ubuntu 20.04+
• cmake >= 3.29
• CUDA >= 12.6
• PyTorch Python 3.9.18
• PyTorch 2.4.0
• NVIDIA Nsight Compute
• one H100 PCIe GPU and one NVIDIA RTX5090 GPU.

2. Environment Setup.

Conda environments need to be set up on machines with H100 PCIe and RTX5090 GPUs following the steps below.

2.1 Install via Conda.

• 2.1.1 Install conda on system. (Toturial).
• 2.1.2 Create a conda environment:

conda create -n env_name python=3.9

• 2.1.3 Install PyTorch (Toturial):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

3. Install FastInfer in kernel benchmark.

cd $FastInfer_HOME/kernel_benchmark/
source init_env

• Build Sputnik.

cd $FastInfer_HOME/third_party/
source build_sputnik.sh

• Build SparTA.

cd $FastInfer_HOME/third_party/
source preparse_cusparselt.sh

The libSpMM_API.so and SpMM_API.cuh will be available for easy integration after:

cd $FastInfer_HOME/kernel_benchmark/
# Choose the target GPU explicitly:
#   h100    : build with the H100 source path and enable WGMMA/v7 kernels.
#   rtx5090 : build with the RTX 5090 source path and disable WGMMA/v7 kernels.
source myinstall.sh h100
# or
source myinstall.sh rtx5090

4. Running FastInfer in kernel benchmark.

For a quick check with a single N value, an N filter is provided in launch.py to benchmark only the specified N value.

# Run the kernel benchmark and baselines from the test directory.
cd $FastInfer_HOME/kernel_benchmark/test
source init_env
nohup python launch.py > logs/launch_log &  //FastInfer-v1,v2,FlashLLM_v1,SpInfer,CuBlas_TC
nohup python launch_sparta.py > logs/sparta_log & 
nohup python launch_cusparse.py > logs/cusparse_log &
nohup python launch_sputnik.py > logs/sputnik_log &

4.1 Reproduce Figure 11

Check the raw throughput CSV files in $FastInfer_HOME/kernel_benchmark/result/kernel/. Then, run all_process.py to generate the merged and speedup CSV files, and run plot.py to reproduce Figure 11.

cd $FastInfer_HOME/kernel_benchmark/result/kernel
python all_process.py
python plot.py

4.2 Reproduce Figure 12

cd $FastInfer_HOME/kernel_benchmark/result/ablation_study
python plot.py

4.3 Reproduce Table 1

Profiling of micro-architectural metrics for sparse kernels. Check the profile_Qwen.ncu-rep using Nsight Compute.

cd $FastInfer_HOME/kernel_benchmark
/usr/local/cuda-12.6/bin/ncu --export ./profile_Qwen ./spmm_test 5120 17408 8 60 5 0 #M,K,N,Sparsity,SplitK,CUDA_VISIBLE_DEVICES

5. Running End-to-end model.

5.1 Building

Build FasterTransformer with the FastInfer integration. Start from a clean FasterTransformer-main tree, apply the FastInfer patch, and then build it with the H100 architecture flag.

cd $FastInfer_HOME/third_party
source init_env
cd ./FasterTransformer-main

# Apply the FastInfer changes to the clean FasterTransformer source tree.
patch -p1 < ../ft_fastinfer_final.patch

# Build FasterTransformer with FastInfer enabled.
mkdir -p build
cd build
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DFastInfer=ON -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

For other end-to-end baselines, keep the same build directory workflow and replace the CMake command with one of the following:

# Standard FasterTransformer: use cuBLAS for all MatMuls.
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DFLASH_LLM=OFF -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

# FasterTransformer with Flash-LLM.
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DFLASH_LLM=ON -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

# FasterTransformer with SpInfer.
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DSpInfer=ON -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

CUDA 12.8 compatibility

When building any FasterTransformer variant with CUDA 12.8, add the following standard-library headers to the corresponding source files:

• quantization_int8_kernels.cu: #include <cstdio>
• quantize_weight.cu: #include <cstdio>
• add_bias_transpose_kernels.cu: #include <cstdint>

5.2 Downloading & Converting OPT models

The following commands use opt-30b as an example. Replace opt-30b with the target OPT model name if you evaluate a different OPT model size.

Download the Hugging Face checkpoint:

cd $FastInfer_HOME/end2end_inference/models
git lfs install
git clone https://huggingface.co/facebook/opt-30b
cd opt-30b
git lfs pull --include="pytorch_model*"

Convert the PyTorch checkpoint to the FasterTransformer format:

cd $FastInfer_HOME/end2end_inference/ft_tools
python huggingface_opt_convert_Phase1.py \
    -i $FastInfer_HOME/end2end_inference/models/opt-30b \
    -o $FastInfer_HOME/end2end_inference/models/opt-30b/c-model \
    -i_g 1 \
    -weight_data_type fp16 \
    -p 64

Here, -i_g is the tensor-parallel GPU number used for inference, and -p is the number of CPU threads used during conversion. Keep -i_g consistent with the GPU count used later by mpirun.

Run the Phase 2 preprocessing script to generate the sparse model files. The scripts below correspond to 1, 2, and 4 GPU tensor-parallel settings:

cd $FastInfer_HOME/end2end_inference/ft_tools

check SplitKDict support for OPT models in huggingface_opt_convert_Phase2_X.py

# 1 GPU
bash prepare.sh

# 2 GPUs
bash prepare-2.sh

# 4 GPUs
bash prepare-4.sh

To change the sparsity ratio, modify p in the corresponding Phase 2 script, for example huggingface_opt_convert_Phase2_FastInfer.py, huggingface_opt_convert_Phase2_flashllm.py, or huggingface_opt_convert_Phase2_spinfer.py. Setting p=0.3 keeps 30% of the weights and produces 70% sparsity.

5.3 Configuration

Initialize the environment.

source init_env
echo $FastInfer_HOME

Before running inference, update the FasterTransformer config file under:

$FastInfer_HOME/third_party/FasterTransformer-main/examples/cpp/multi_gpu_gpt/

The provided config files are named by request batch size, for example gpt_config_8.ini, gpt_config_16.ini, gpt_config_32.ini, and gpt_config_64.ini. You can edit one of them directly, or copy it to a new file before changing the fields below.

The key fields are:

model_name=opt_30B
tensor_para_size=1
model_dir=$FastInfer_HOME/end2end_inference/models/opt-30b/c-model/1-gpu-FastInfer
request_batch_size=8

Please keep the following settings consistent:

• model_name: use the model size you converted, such as opt_30B.
• tensor_para_size: set this to the same GPU number used by -i_g during conversion and by mpirun -n during inference.
• model_dir: point this to the converted model directory generated in Section 5.2.
• request_batch_size: match the config file or the batch size you want to evaluate.

Choose the model_dir suffix according to the backend:

# FastInfer
model_dir=$FastInfer_HOME/end2end_inference/models/opt-30b/c-model/1-gpu-FastInfer

# Flash-LLM
model_dir=$FastInfer_HOME/end2end_inference/models/opt-30b/c-model/1-gpu-flashllm

# SpInfer
model_dir=$FastInfer_HOME/end2end_inference/models/opt-30b/c-model/1-gpu-spinfer

# Standard FasterTransformer
model_dir=$FastInfer_HOME/end2end_inference/models/opt-30b/c-model/1-gpu

For 2-GPU or 4-GPU inference, replace 1-gpu-* with 2-gpu-* or 4-gpu-*, and set tensor_para_size to 2 or 4.

5.4 Running Inference

Run the FasterTransformer end-to-end inference for FastInfer, SpInfer, or Flash-LLM from the patched source tree:

# Replace X with fastinfer, spinfer, or flashllm
cd $FastInfer_HOME/third_party/FasterTransformer\_X
    bash run_all.sh

run_all.sh runs the configured batch sizes in sequence. By default, it uses gpt_config_8.ini, gpt_config_16.ini, and gpt_config_32.ini, together with examples/cpp/multi_gpu_gpt/start_ids_64.csv.

To run a single config manually, use:

#The `-n` value should match `tensor_para_size` in the selected config file. For example, `-n 1` runs inference on a single GPU.
CUDA_VISIBLE_DEVICES=0 mpirun -n 1 --allow-run-as-root -x CUDA_VISIBLE_DEVICES -x LD_LIBRARY_PATH \
    ./build/bin/multi_gpu_gpt_example \
    ./examples/cpp/multi_gpu_gpt/gpt_config_8.ini \
    ./examples/cpp/multi_gpu_gpt/start_ids_64.csv

For DeepSpeed, record its measured latency with the same CSV schema and save it as end2end_times_deepspeed.csv.

cd $FastInfer_HOME/end2end_inference/ds_scripts
pip install -r requirements.txt
bash run_all.sh

5.5 Reproduce Figure 14

The result directory is:

#For example, for OPT-30B:
$FastInfer_HOME/end2end_inference/result/opt-30b

Each baseline should provide one CSV file with the same format:

batch_size,config,time_ms
8,./examples/cpp/multi_gpu_gpt/gpt_config_8.ini,7316.60
16,./examples/cpp/multi_gpu_gpt/gpt_config_16.ini,8277.65
32,./examples/cpp/multi_gpu_gpt/gpt_config_32.ini,10191.47

The expected file names are:

end2end_times_fastinfer.csv
end2end_times_spinfer.csv
end2end_times_flashllm.csv
end2end_times_deepspeed.csv
end2end_times_fastertransformer.csv

Generate Figure 14:

cd $FastInfer_HOME/end2end_inference/result/opt-30b
python plot.py

The output figure is:

$FastInfer_HOME/end2end_inference/result/opt-30b/opt_30b_50.png