Quick Start Guide for OpenAI API Chat Server

Test the OpenAI API Server (fastapi)

prepare: pip install fastapi uvicorn openai

start server python ./dash-infer/examples/api_server/fastapi/fastapi-server.py

user may change the parameter by check fastapi-server.py -h

After sever start, server will print some log like:

INFO:     Started server process [4898]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

test with openai client python ./dash-infer/examples/api_server/fastapi/openai-client.py

This client will call with openai client with streaming and block mode.

Start OpenAI Server with Docker (fastchat)

We have provide a Docker image to start OpenAI server. This example demonstrates how to use Docker to run DashInfer as an inference engine, providing OpenAI API endpoints.

docker run \
    --shm-size=8g \
    --network=host \
    --ipc=host \
    --gpus=all \
    -v=<host_path_to_your_model>:<container_path_to_your_model> \
    dashinfer/fschat_ubuntu_cuda:v2.0 \
    -h <host> -p <port> -m -- --model-path <container_path_to_your_model> --device-list <device_list>

<host_path_to_your_model>: Path to your model on the host
<container_path_to_your_model>: Path where the model is mounted in the container
<device_list>: List of devices supported by the model, e.g., 0,1
-h: The listening address of the API server
-p: The listening port of the API server
-m: Use Modelscope to download the model
--model-path: Path for loading or downloading the model
--device-list: List of CUDA devices used to run the model

For example:

docker run  \
    --shm-size=8g \
    --network=host \
    --ipc=host \
    --gpus=all \
    -v=/mnt/models/modelscope/hub/qwen/Qwen2.5-7B-Instruct:/models/qwen/Qwen2.5-7B-Instruct \
    dashinfer/fschat_ubuntu_cuda:v2.0 \
    -h 127.0.0.1 -p 8088 -m -- --model-path /models/qwen/Qwen2.5-7B-Instruct --device-list 0

You can also build you owner fastchat Docker image by modifying the Docker file scripts/docker/fschat_ubuntu_cuda.Dockerfile.

Testing the OpenAI API Server (fastchat)

Testing with OpenAI SDK

In examples/api_server/fschat/openai-client.py, the official OpenAI SDK is used to test the API server.

python examples/api_server/fschat/openai-client.py

Testing with curl

Assuming the OpenAI Server has been started and the port number is 8088, you can use the following command:

curl http://127.0.0.1:8088/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2-7B-Instruct",
    "messages": [{"role": "user", "content": "Hello! What is your name?"}]
  }'