server: docs: --threads and --threads, --ubatch-size, --log-disable
#6254
Conversation
| @@ -26,7 +26,8 @@ The project is under active development, and we are [looking for feedback and co | |||
| - `-ngl N`, `--n-gpu-layers N`: When compiled with appropriate support (currently CLBlast or cuBLAS), this option allows offloading some layers to the GPU for computation. Generally results in increased performance. | |||
| - `-mg i, --main-gpu i`: When using multiple GPUs this option controls which GPU is used for small tensors for which the overhead of splitting the computation across all GPUs is not worthwhile. The GPU in question will use slightly more VRAM to store a scratch buffer for temporary results. By default GPU 0 is used. Requires cuBLAS. | |||
| - `-ts SPLIT, --tensor-split SPLIT`: When using multiple GPUs this option controls how large tensors should be split across all GPUs. `SPLIT` is a comma-separated list of non-negative values that assigns the proportion of data that each GPU should get in order. For example, "3,2" will assign 60% of the data to GPU 0 and 40% to GPU 1. By default the data is split in proportion to VRAM but this may not be optimal for performance. Requires cuBLAS. | |||
| - `-b N`, `--batch-size N`: Set the batch size for prompt processing. Default: `512`. | |||
| - `-b N`, `--batch-size N`: Set the batch size for prompt processing. Default: `2048`. | |||
| - `-ub N`, `--ubatch-size N`: physical maximum batch size. Default: `512`. | |||
There was a problem hiding this comment.
Should we also make it clear that ubatch should be enough when using for embeddings?
There was a problem hiding this comment.
Yes, there is no advantage to increasing n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.
There was a problem hiding this comment.
@ngxson I think it is better to add this check this in server.cpp. i will create an issue to trace it, we will implement it later on.
There was a problem hiding this comment.
The embeddings from multiple slots can go in a single batch. For example with n_batch = 2048 and n_ubatch = 512 we can process 4 full slots in one go
There was a problem hiding this comment.
It could be implemented, but the batch splitting code does not take this into account. llama_decode will just fail if n_tokens > n_ubatch.
There was a problem hiding this comment.
We can move this discussion in ?
There was a problem hiding this comment.
Ah yes sorry, ignore my comment
Update
server/README.mdwith actual params supported.