Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backend : offload large batches to GPU #6083

Merged
merged 9 commits into from Mar 18, 2024
Merged

backend : offload large batches to GPU #6083

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5b6b4ac
backend : offload large batches to GPU
slaren Mar 14, 2024
c2dba04
fix hip
slaren Mar 15, 2024
3a77442
code cleanup
slaren Mar 15, 2024
c0fe629
fix CUDA split buffers
slaren Mar 15, 2024
8e717e8
Update ggml-backend-impl.h
slaren Mar 16, 2024
9cba8a1
cuda : fix memset without set_device
slaren Mar 16, 2024
9809075
imatrix : remove sched affix from weight names
slaren Mar 16, 2024
0661e6a
sched : add a new split if the current one has too many inputs
slaren Mar 16, 2024
cc9299c
update backends
slaren Mar 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Next Next commit
cuda : fix memset without set_device
  • Loading branch information
@slaren
slaren committed Mar 16, 2024
commit 9cba8a183d147158009c6b57803378aa9962c521
1 change: 1 addition & 0 deletions ggml-cuda.cu
View file
Open in desktop
slaren marked this conversation as resolved.
Show resolved Hide resolved
Expand Up @@ -10618,6 +10618,7 @@ GGML_CALL static void ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer_t
size_t padded_size = ggml_backend_buft_get_alloc_size(buffer->buft, tensor);

if (padded_size > original_size && tensor->view_src == nullptr) {
ggml_cuda_set_device(ctx->device);
CUDA_CHECK(cudaMemset((char *)tensor->data + original_size, 0, padded_size - original_size));
}
}
Expand Down