Environment Variable Usage
This section describes the definition of the DashInfer environment variables and their function.
Memory Mangament
EnvVar Name |
Describe |
Type |
Default |
Options |
|---|---|---|---|---|
|
Use BFC Allocator or raw cudaMalloc API for management of CUDA Device memory. |
bool |
|
ON - Enable BFC Allocator OFF - Disable BFC Allocator |
|
The max ratio of device memory that will be managemented by BFC Allocator. |
float |
|
float value between (0.0,1.0] |
|
The amount of GPU memory that cannot be used by the BFC allocator, typically the sum of GPU memory occupied by PyTorch, CUDA driver, default context, etc. |
int |
|
Formula for the actual GPU memory allocated by the BFC Allocator on each GPU:
|
|
DashInfer will set CPU_CACHE_RATIO * 100% of the current remaining CPU memory for kv cache storage, and when CPU_CACHE_RATIO=0, no CPU memory is used to store kv cache. |
float |
|
float value between [0.0, 1.0] |
Logging
EnvVar Name |
Describe |
Type |
Default |
Options |
|---|---|---|---|---|
|
Whether logging the generation and context step detailed time in different phase. |
int |
|
|
|
Whether print output token in log. |
int |
|
|
|
The threshold control for printing statistical log when text generation. |
int |
|
In second, should be greater than 0. |
Engine Behavior
EnvVar Name |
Describe |
Type |
Default |
Options |
|---|---|---|---|---|
|
Use the same sampler as vllm and PyTorch. The generation speed may decrease by 5%-10%. |
int |
|
which provides the same distribution, but not exactly the same value. |
|
Threshold for enable Flash Attention do context attention calculation. Flash Attention will be used if context length is greater than this threshold. |
int |
|
int value between (0, int64_max). |
|
Disable warm up step when model is start up. |
int |
|
|