vLLM Integration

IntelliNode provides seamless integration with self-hosted vLLM models.

Supported Models

Examples of commonly used vLLM models include:

Model Name	Description
meta-llama/Llama-3.1-8B-Instruct	Llama3 instruct model
deepseek-ai/DeepSeek-R1-Distill-Llama-8B	DeepSeek distilled model
BAAI/bge-small-en-v1.5	Embedding model

Note: vLLM supports many other models hosted locally or remotely.

Setup & Usage

Chat Completion

Step 1: Import required modules.

const { Chatbot, VLLMInput, SupportedChatModels } = require('intellinode');

Step 2: Set your vLLM server URL.

const vllmUrl = 'http://localhost:8000';
const chatbot = new Chatbot(null, SupportedChatModels.VLLM, null, { baseUrl: vllmUrl });

Step 3: Create input and add user message.

const input = new VLLMInput('You are a helpful assistant.', {
  model: 'meta-llama/Llama-3.1-8B-Instruct',
  maxTokens: 100,
  temperature: 0.7
});

input.addUserMessage('What is machine learning?');

Step 4: Get response from your chatbot.

const response = await chatbot.chat(input);
console.log('Chatbot response:', response);

DeepSeek Model Example

const deepseekInput = new VLLMInput('You are a helpful assistant.', {
  model: 'deepseek-ai/DeepSeek-R1-Distill-Llama-8B',
  maxTokens: 150,
  temperature: 0.6
});

deepseekInput.addUserMessage('Explain quantum computing briefly.');

const response = await chatbot.chat(deepseekInput);
console.log('DeepSeek response:', response);

---

Generating Embeddings with vLLM

IntelliNode makes it simple to generate text embeddings using vLLM-hosted models.

Step 1: Import modules.

const { RemoteEmbedModel, SupportedEmbedModels } = require('intellinode');
const EmbedInput = require('intellinode').EmbedInput;

Step 2: Set your vLLM embedding server URL.

const embedUrl = 'http://localhost:8001';
const embedModel = new RemoteEmbedModel(null, SupportedEmbedModels.VLLM, { baseUrl: embedUrl });

Step 3: Create embedding input.

const input = new EmbedInput({ texts: ['Hello world', 'vLLM embeddings example'] });

Step 4: Generate embeddings.

const embeddings = await embedModel.getEmbeddings(input);
console.log('Embeddings:', embeddings);

---

Chat With Your vLLM Server From the UI

IntelliChat connects to vLLM from the browser, with the same Chatbot behind it:

1. Run IntelliChat locally, as described in Chat UI, so it can reach your server.
2. Open the settings and select vLLM (self-hosted).
3. Enter your server URL, for example http://localhost:8000.
4. Load the served models, or type a model id such as meta-llama/Llama-3.1-8B-Instruct.