New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add zstd support #2856
Comments
Also previously discussed in #711 |
Thank you for your request. As @lurch pointed out it was previously discussed about a year ago, so we can at least see if the situation got better wrt dealing with 7-zip |
@sedlund Out of curiosity, are you aware of disk images that are being distributed in zstd format? |
@lurch I haven't seen anything offered as of yet - I suppose its a bit chicken and egg now. I use it for creating SD backup images compressed on the device and then streamed over the network to its final destination. I'm sure you've seen the benchmarks, Zstd is far and away faster in compressing on small devices, 7zip itself doesn't have Zstd support. There is a fork that includes it: I'm not sure why you'd need 7zip to implement Zstd though. I would think it would be implemented similarly to how lzma was in Etcher. |
A quick search finds https://www.npmjs.com/package/zstd which was last published 4 years ago and https://www.npmjs.com/package/node-zstandard which was last published 3 years ago 🙁 |
@lurch I guess a lot of iteration is not required when you only need to implement 2 methods to hook to an external lib - compress and decompress eh? 😃 |
Sorry, I was indirectly referring to your "using an older Zstd 1.2 - it has received many large performance increases since then" comment - have these "performance increases" been added before or after these node packages were last published? 🤷♂️ |
I've been looking into this because of the speed of ZSTD's decompression, combined with its good compression ratio, which I think offers a uniquely useful algorithm for compressing disk images and the like. It looks like the main blocker is the age of existing ZSTD node modules, the lack of a module that implements a node Transform stream, and the small Venn diagram of people who know Node and C++ well enough to make a native module implementing this interface. Fortunately, compiling libzstd to WebAssembly is pretty straightforward with Emscripten, at which point we can just call exported methods directly from JavaScript. I'm no JS guru, but I know my way around Emscripten, and this is an interesting project to me, so I guess we'll see where it goes. |
Is there any news on this topic? |
I would greatly appreciate this feature as well! Thanks! |
I am producing |
Really large images, like operating systems, are starting to use zstd more. That's what brought me here. So to @lurch's question "are you aware of disk images that are being distributed in zstd?" the increasing answer is yes and interest seems to be trending up. I hadn't heard of ZSTD before a year ago, so I don't perceive this as a red hot issue, but it must be growing. In my case, I just wanted a streaming decompress so I don't have to read the big file to write a bigger file and then use Etcher to safely copy that. I'd be perfectly happy for that 'bigger' image to exist only in temporary memory as it was streamed to the USB card. If libzstd doesn't float your boat, cloudflare has cleaned up the really really old zlib code, branched it, and added zstd. See https://indico.cern.ch/event/695984/contributions/2872933/attachments/1590457/2516802/ZSTD_and_ZLIB_Updates_-_January_20186.pdf While I'm here, thank you for a great cross-platform utility. I know those are hard to make and Etcher rocks. |
Chiming in here - I use zstd heavily in my image workflow for SBC distro and ChromeOS images. Zstd support would save me a lot of time, xz is space efficient but has crappy compress/decompress compute and time efficiency. |
would be nice to add to etcher pro! |
[mpous] This has attached https://jel.ly.fish/d5aa35db-a306-4791-b408-4817d247a383 |
I agree. Since #2048 was closed as uncompleted, |
Yocto appears to be moving to ZStandard for compressing WIC images, fwiw. |
Please add support for zstd compressed images.
The text was updated successfully, but these errors were encountered: