> For the complete documentation index, see [llms.txt](https://docs.sesterce.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.sesterce.com/tutorials/expose-ai-model-from-hugging-face-using-vllm.md).

# Expose AI model from Hugging Face using vLLM

You want to expose to the world an API endpoint to allow your end-users to access an AI Model from Hugging Face? This tutorial is for you!

## Choose your VM offer

The first step is to choose the VM offer tailored to your needs on [**Sesterce Cloud**](https://cloud.sesterce.com/compute). The choice of the instance will depend on several factors, such as the size of your model and the number of end-users.

<figure><img src="https://3376774032-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoExYATuECEyEKGJ4cD8X%2Fuploads%2FlnP3V3nUJmmxj5OfDNzP%2FScreenshot%202025-05-06%20at%2018.00.21.png?alt=media&amp;token=54b95d76-0386-4249-8557-527d9365e452" alt=""><figcaption></figcaption></figure>

You can consider following examples according to your use-case:

**Small Scale (1-10 simultaneous users)**

* 7B params: L4 (24GB VRAM)
* 13B params: L40S (48GB VRAM)
* Example: Mistral 7B with 8 users in 4K context = 16GB + (8 × 2GB) = 32GB VRAM

**Medium Scale (10-50 users)**

* 7B params : 2-4x L40S in parallel
* 13B params: 4x L40S or H100
* Recommended configuration: Load balancing between multiple GPUs

**Large Scale (50+ users)**

* 7B params: 8x L40S or 2x H100
* 13B+ params: H100 multi-GPU
* Use quantization techniques (INT8/INT4)

## Set-up your instance

Well done, you are now able to configure your VM! Please find here [**the steps required to do it**](https://docs.sesterce.com/compute-instances/configure-your-compute-instance.md). In "Images" section, choose vLLM option. The instance launch usually takes around 5 minutes.

<figure><img src="https://3376774032-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoExYATuECEyEKGJ4cD8X%2Fuploads%2FJstIzDYeBnHAdNd9wYT9%2FScreenshot%202025-05-07%20at%2009.10.14.png?alt=media&amp;token=bcade5aa-8a37-46bd-93e4-7854d5dee0ee" alt=""><figcaption></figcaption></figure>

## Connect to your instance

When your VM instance is launched, you'll be able to get ssh command to connect into it, like `ssh sesterce@<IP_MACHINE>`.&#x20;

{% hint style="info" %}
**Pay attention:** the docker pull is running! Wait until it's finished to type the following command.
{% endhint %}

When the docker pull running is finished, use the following command:

{% code collapsedlinecount="10" %}

```
docker run -d -e "HF_TOKEN=<HUGGING_FACE_TOKEN>" --runtime nvidia --gpus all --net=host --ipc=host vllm/vllm-openai --model "<MODEL_ID>"
```

{% endcode %}

Before running the command, replace \<HUGGING\_FACE\_TOKEN> with your Hugging Face token and \<MODEL\_ID> with your model identifier. Keep the quotes around each value.

<figure><img src="https://3376774032-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoExYATuECEyEKGJ4cD8X%2Fuploads%2FMr495SdynAs2c7z3yN23%2FScreenshot%202025-05-07%20at%2009.17.27.png?alt=media&amp;token=f9e0d7a6-2341-410f-9110-52a618d79331" alt=""><figcaption><p>Create token from Hugging Face</p></figcaption></figure>

<figure><img src="https://3376774032-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoExYATuECEyEKGJ4cD8X%2Fuploads%2F2dVDJGvztaTDSbHret8b%2FScreenshot%202025-05-07%20at%2009.22.27.png?alt=media&amp;token=20ee480d-51ee-419a-9b07-e9b3763db3f5" alt=""><figcaption><p>Get your Model ID from Hugging Face</p></figcaption></figure>

## Use your model

Once the container is running and the model has finished loading, send a request to port 8000. Replace \<IP\_MACHINE> with your instance IP address and \<MODEL\_ID> with your model identifier; keep the URL in quotes.

{% code collapsedlinecount="10" %}

```
curl -X POST "http://<IP_MACHINE>:8000/v1/completions" -H "Content-Type: application/json" -d '{"model": "<MODEL_ID>","prompt": "Hello world","max_tokens": 50,"temperature": 0}'
```

{% endcode %}

List the available models at `http://<IP_MACHINE>:8000/v1/models`. Replace \<IP\_MACHINE> with your instance IP address.

<figure><img src="https://3376774032-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoExYATuECEyEKGJ4cD8X%2Fuploads%2FXcxEBj7RQtzwKDkwIoYO%2Fimage%20(7).png?alt=media&amp;token=0a8dcef2-10af-4d35-ab5d-f0f5fda725a7" alt=""><figcaption></figcaption></figure>

## Common errors

### Instance RAM insufficient

Make sure you choose an instance with sufficient RAM to run your model.

### Container not running yet

Check both running and stopped containers with the following command:

{% code collapsedlinecount="10" %}

```
docker ps -a
```

{% endcode %}

* If your container is absent, check whether the docker run command succeeded. If it has exited, inspect its logs for the startup error.
* Copy the container name or ID from the list, replace \<CONTAINER\_NAME> below, and inspect its logs. A running container may still be loading the model.

{% code collapsedlinecount="10" %}

```
docker logs "<CONTAINER_NAME>"
```

{% endcode %}

### Port used or unavailable

Check whether a service is listening on the default port 8000. If you configured another port, replace 8000 in the command:

{% code collapsedlinecount="10" %}

```
ss -tuln | grep ":8000[[:space:]]"
```

{% endcode %}
