Instructions to use nebius/MLP-Speculator-Llama-3.1-8B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nebius/MLP-Speculator-Llama-3.1-8B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nebius/MLP-Speculator-Llama-3.1-8B-Instruct")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("nebius/MLP-Speculator-Llama-3.1-8B-Instruct", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nebius/MLP-Speculator-Llama-3.1-8B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nebius/MLP-Speculator-Llama-3.1-8B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nebius/MLP-Speculator-Llama-3.1-8B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/nebius/MLP-Speculator-Llama-3.1-8B-Instruct

SGLang

How to use nebius/MLP-Speculator-Llama-3.1-8B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nebius/MLP-Speculator-Llama-3.1-8B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nebius/MLP-Speculator-Llama-3.1-8B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nebius/MLP-Speculator-Llama-3.1-8B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nebius/MLP-Speculator-Llama-3.1-8B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use nebius/MLP-Speculator-Llama-3.1-8B-Instruct with Docker Model Runner:
```
docker model run hf.co/nebius/MLP-Speculator-Llama-3.1-8B-Instruct
```

Model Description

This is an MLP-Speculator draft model for Llama-3.1-8B-Instruct, trained from scratch using LK losses — training objectives that directly target acceptance rate rather than using KL divergence as a proxy.

Training Details

Base model: meta-llama/Llama-3.1-8B-Instruct
Draft architecture: MLP-Speculator
Training data: Infinity-Instruct-0625 with Llama-3.1-8B generated responses
Training objective: Hybrid LK loss with adaptive λ scheduling (η=3)
Training: 10 epochs from random initialization
Draft length: K = 6 speculative tokens

Performance

Average acceptance length (τ) measured across MT-bench, HumanEval, and GSM8K with K = 6:

Configuration	Temperature = 0	Temperature = 1
MLP-Speculator + KL	2.43	2.15
MLP-Speculator + LK (ours)	2.59	2.33

Note: Earlier vLLM versions sampled draft tokens greedily regardless of temperature, which underestimated acceptance rates at temperature > 0. Stochastic draft sampling was introduced in v0.18.0, and from v0.21.0 it can be enabled via speculative_config using rejection_sample_method and draft_sample_method. The acceptance metrics reported above were measured under standard rejection sampling and are reproducible with the configuration below.

Usage with vLLM

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    speculative_config={
        "method": "mlp_speculator",
        "model": "nebius/MLP-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 6,
        "rejection_sample_method": "standard",
        "draft_sample_method": "gumbel",
    },
)

sampling_params = SamplingParams(temperature=0.7)
outputs = llm.generate(["Explain speculative decoding in simple terms."], sampling_params)

License

CC BY 4.0

This model was trained using outputs from meta-llama/Llama-3.1-8B-Instruct. Use of this model is additionally subject to the Llama 3.1 Community License Agreement.

Citation

@misc{samarin2026lklosses,
  title     = {LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding},
  author    = {Alexander Samarin and Sergei Krutikov and Anton Shevtsov and Sergei Skvortsov and Filipp Fisin and Alexander Golubev},
  year      = {2026},
  eprint    = {2602.23881},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url       = {https://arxiv.org/abs/2602.23881}
}