If you have any suggestions to improve the organization of the files, the naming scheme or anything else, please let me know, I am sure it could be improved.

Other than that, the only thing left to do is fix the HIP and Makefile builds.

I think previously we ought to unify all functions into one file #3965 (comment), now the design has changed? Could you share any more details?

There was a brief discussion about this in #5434. The compilation time of the CUDA backend was increasing to the point that it was complicating working on it. Personally, I also prefer to work on multiple smaller files than in one large file, but I don't know if everybody agrees about that.

There was a brief discussion about this in #5434. The compilation time of the CUDA backend was increasing to the point that it was complicating working on it. Personally, I also prefer to work on multiple smaller files than in one large file, but I don't know if everybody agrees about that.

Will you summary it in a document, for example, "guidelines for adding new backends"? I see some backends are WIP like #6035 and following new designs might be easier.

I apologize for nagging but LLAMA_CUDA_F16 is broken again.

ggml-cuda/dmmv.cu(768): error: identifier "to_fp16_cuda_t" is undefined
          const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src1->type);
                ^

ggml-cuda/dmmv.cu(768): error: identifier "ggml_get_to_fp16_cuda" is undefined
          const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src1->type);
                                              ^

2 errors detected in the compilation of "ggml-cuda/dmmv.cu".

Should be fixed in #6298

Considering the extremely long ggml-cuda.cu compilation times I was pleasantly surprised by this refactoring. Until I had to adapt PR #6302 to the refactoring: touching vecdotq.cuh, which one needs to do all the time when developing new quants or optimizing dot product kernels, leads to the very same extremely long compilation as before (this time of mmq.cu).

The build time is dominated by mmvq.cu, and to a lesser degree by mmq.cu, so as you say, this will not help much if you are changing these files. It should be possible to split each quant type to a different file as well to improve this.