Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
[BREAKING CHANGE] Ignore added_tokens (both special and not) in the decoder
#1513
opened Apr 24, 2024 by
Narsil
Loading…
change conditional compilation for regex libraries
#1510
opened Apr 23, 2024 by
semaraugusto
Loading…
Add
.editorconfig and rustfmt.toml for Consistent Code Formatting
#1508
opened Apr 21, 2024 by
tal7aouy
Loading…
corrected typo in the documentations for pre-tokenizers
#1503
opened Apr 17, 2024 by
GorkaUrbizu
Loading…
Fix unsoundness in
tokenizers::utils::parallelism
#1492
opened Apr 12, 2024 by
albertsgarde
Loading…
Bump express from 4.18.1 to 4.19.2 in /tokenizers/examples/unstable_wasm/www
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
Stale
#1479
opened Mar 28, 2024 by
dependabot
bot
Loading…
Add normalization option to Chinese characters (using OpenCC) and separate symbols from merging
#473
opened Oct 19, 2020 by
ecchochan
Loading…
remove use of parallel iterators except in batch methods
#308
opened Jun 17, 2020 by
epwalsh
Loading…
ProTip!
Mix and match filters to narrow down what you’re looking for.