Please confirm you are human
This browser or connection looks automated. Press and continuously hold the control for 3 seconds to enable Google-hosted web results and, when separately allowed, AI-assisted answers.
A successful check enables 100 search requests. Interactive access does not authorize scraping, systematic collection, or reuse of search output.
News
kda_metadata
3+ week, 4+ day ago (57+ words) vLLM Kimi-K3 specialization of GDN attention metadata. The request classification and cudagraph staging intentionally mirror GDNAttentionMetadataBuilder. Kimi-K3 builds the metadata required by its prefill KDA kernel internally, so this builder omits the shared FLA chunk metadata construction. For Kimi-K3 speculative…...
fused_topk_bias_router
7+ mon, 5+ day ago (21+ words) vLLM Router using fused top-k with e_score_correction_bias. Compute routing using fused top-k with bias....
moonvit
9+ mon, 2+ week ago (284+ words) Build packed per-token positional embeddings for a list of grids. whether to use bias in linear layer. precomputed positional embeddings of shape (L, Cout). When provided, grid_hw is unused and the CUDA-graph-incompatible interpolation in self.pos_emb is skipped. Optional precomputed scalar tensor....
Production Metrics
9+ mon, 2+ week ago (92+ words) vLLM exposes a number of metrics that can be used to monitor the health of the system. These metrics are exposed via the /metrics endpoint on the vLLM OpenAI compatible API server. You can start the server using Python, or…...
server_utils
4+ day, 7+ hour ago (47+ words) vLLM Get the uvicorn log config based on the provided arguments. Priority: 1. If log_config_file is specified, use it 2. If disable_access_log_for_endpoints is specified, create a config with the access log filter 3. Otherwise, return None (use uvicorn defaults)...
protocol
9+ mon, 2+ week ago (48+ words) Pydantic models for Anthropic API protocol Content block in message Delta for streaming responses Error structure for Anthropic API Error response structure for Anthropic API JSON output format configuration Anthropic Messages API request Anthropic Messages API response Configuration options for…...
glm_ocr
6+ mon, 3+ week ago (10+ words) vLLM Inference-only GLM-OCR model compatible with HuggingFace weights....
audio
1+ week, 1+ day ago (22+ words) vLLM _forward_speech_encoder(mel_features, input_seq_lens, audio_sample_lens) ¶ Run the eager speech encoder without server-side slicing/batching....
dots_tool_parser
1+ week, 1+ day ago (55+ words) vLLM Parse Dots tool calls in their XML wrapper format. The canonical body contains one or more invoke elements:: A JSON object with name and arguments is also accepted as a fallback. Multiple wrapper blocks and multiple invokes per block…...
layernorm
1+ week, 3+ day ago (33+ words) vLLM _fused_add_rms_norm(x, x_residual, weight, epsilon, variance_size=None) ¶ Fused add and weighted root-mean-square layer normalization _rms_norm(x, weight, epsilon, variance_size=None) ¶ Weighted root-mean-square layer normalization...