perplexity : support using multiple sequences to allow larger batch sizes #5946
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
compilade
reviewed
Mar 9, 2024
compilade
reviewed
Mar 9, 2024
ggerganov
approved these changes
Mar 9, 2024
|
This in theory should applied to imatrix as well |
|
Probably won't help with imatrix unless using very small context sizes. As it is, imatrix will also not benefit from pipeline parallelism because reading the activations forces a synchronization. |
hazelnutcloud
pushed a commit
to hazelnutcloud/llama.cpp
that referenced
this pull request
Mar 10, 2024
…izes (ggerganov#5946) * perplexity : support using multiple sequences to allow larger batch sizes ggml-ci * set cparams.n_parallel to the number of sequences * print tested n_ctx, add assert
NeoZhangJianyu
pushed a commit
to NeoZhangJianyu/llama.cpp
that referenced
this pull request
Mar 12, 2024
…izes (ggerganov#5946) * perplexity : support using multiple sequences to allow larger batch sizes ggml-ci * set cparams.n_parallel to the number of sequences * print tested n_ctx, add assert
jordankanter
pushed a commit
to jordankanter/llama.cpp
that referenced
this pull request
Mar 13, 2024
…izes (ggerganov#5946) * perplexity : support using multiple sequences to allow larger batch sizes ggml-ci * set cparams.n_parallel to the number of sequences * print tested n_ctx, add assert
hodlen
pushed a commit
to hodlen/llama.cpp
that referenced
this pull request
Apr 1, 2024
…izes (ggerganov#5946) * perplexity : support using multiple sequences to allow larger batch sizes ggml-ci * set cparams.n_parallel to the number of sequences * print tested n_ctx, add assert
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allows increasing the batch size with
perplexity. The batch size must be a multiple ofn_ctx.There is a small improvement to the performance since the batching API allows extracting only the logits that are actually used, which reduces the amount of data that needs to be copied back from the GPU, and increasing the batch size can help with quantized models when using a small context, but mainly the goal is to allow using larger batch sizes with pipeline parallelism when using multiple GPUs.