The last year was mainly one of stability. Working up to
0.23 we had fixed
many outstanding stability risks and it continues to be one of the most
long-serving major versions in a while. In this post you find an overview on
the massive progress made with it, and outstanding issues that you could
contribute towards in 2021.
We also have a logo now:
Developments in 2020
Part of the stability work was the ability to upgrade the crates with decoder
implementations such as
gif without breaking the API of the
This allowed you to silently benefit from the improved
gif decoder as
detailed above and other similar fixes across other decoders and it will also
enable similar updates for
This is directly reflected in dependencies and downloads. While previously
there was a long tail, the dependencies and downloads are now dominantly for
recent versions and thus quicker, using less
memory, more correct and with less panics. (If you’re still using
0.22, you should really consider updating). We’ve also more than doubled
our total downloads over the last year! Around 1_000 (one thousand!) crates now
depend on at least one of the crates under our organization. That’s mind
Despite the API stability, we’ve made a ton of progress!
The list of supported formats has expanded greatly. The formats
avif and several DDS subtypes are now recognized or can be written.
The existing formats also cover more of the specification.
Let’s start with the basics. The
png library now supports decoding of
individual APNG frames. It also now treats all allowed pixel formats properly,
including sixteen bit depth. This all makes it pass the extended
with flying colors. It also gained the ability to encode indexed data directly.
tiff crate has also seen a lot of attention. Throughout the year various
contributors added BigTIFF and basic GeoTIFF support, have expanded the
supported sample formats to 32-bit and 64-bit color depth, and overall
increased the amount of recognized tags. It also decodes embedded modern jpeg
image data. Nevertheless, one particularly sore point remains:
tiff does not
yet support Tile Offsets which would be extremely valuable for access to large
data from medical and geodetic data. We’d be delighted to see contributions for
jpeg-decoder crate now correctly decodes all progressive images. It now
collects ICC and Exif data and makes them available in their raw form. The
implementation exposing these in a general interface within
image is expected
to commence sometime soon.
We’ve tested the new implementations on web data dumps of hundreds of thousands of images from the web—totalling terabytes of data— and decoded the overwhelming majority of them successfully.
gif crate joins
png in being on par with the relevant C implementations
in terms of performance! This was made possible by a combination of
improvements. Firstly, the restructured decoder interface make it possible to
write directly into the output buffer which avoids a few copies. Secondly, the
decompression libraries have been upgraded.
These gains are thanks to a new LZW encoding/decoding library, called
weezl. It significantly improves on lzw crate in performance while
staying 100% safe, and also had an optional
no_std mode. There is integration
async IO-interfaces, behind separate features.
written by the author of this post, is also maintained under the image-rs
Since most of the time in decoding GIF files is spent in LZW decoding, the gains from LZW have naturally translated to gif crate. TIFF files encoded with LZW compression also benefit from this change. The total decoding throughput quadrupled in some instances! The latest versions of gif, tiff and image all take advantage of this new library.
jpeg-decoder crate has also been sped up significantly. Idiomatic
iterator based loops of Rust allow the LLVM compiler to aggressively use auto
vectorization. That is, instead of performing a transformation on single
elements at a time this optimization technique automatically replaces code
operating on elements sequentially (in this case, pixels) with special SIMD
instructions that perform them on multiple elements at a time, with large
speedups of 400% or more. While not yet on-par with handwritten assembly this
can get very close with the right code structure.
Additionally, a number of needless bounds checks have been identified. It was possible to remove them with an iterator-based approach, which improved performance without sacrificing safety. Together, the whole decoding process takes 20%-30% less processor time. Finally, the decoder now makes better use of available parallelism. It now spawns a worker thread for each individual component which substantially reduces wall clock time on multi core systems.
One significant outstanding opportunity at the moment is the suboptimal
implementation of the Huffman decoding process. It takes up a disproportionate
fraction of the overall time and blocks all worker threads. You can find more
jpeg-decoder#155 and the Huffman source code.
There have also been a number of notable new or improved dependent crates. What sets this year apart from previous years is the impression that an increasing fractions of these are end-user applications. And Rust is used a tool for delivering to larger audience than the Rust ecosystem itself, with speed, reliability and productivity.
emulsion: A fast and minimalistic image viewer.
czkawka: Multi functional app to find duplicates, empty folders, similar images etc.
img_hash: A Rust library for calculating perceptual hash values of images.
nannou: A Creative Coding Framework for Rust.
gst-plugins-rs: Rust gstreamer plugins with various image and video options that use
Open ideas for 2021
There are still many outstanding issues and unimplemented improvements. From the point of view of a maintainer, some of the most pressing ones are listed in the next sections, sorted roughly by topic.
Fallbacks and Error logs
The decoders are strict, in many ways. There exist a few fallbacks here and there, but we still occasionally encounter a technically invalid image that is successfully decoder by other tools. (Some formats take more liberties than others). The source of this trouble is that being too lenient in decoders leads to an ossification of formats where particular non-standard behavior by the most popular encoders is de-facto standardized without appearing in any specification. For many older formats that ship has, however, long sailed. We should thus accept the popular interpretation (e.g. ImageMagick) of such extensions.
The issue goes a bit farther than this. It would be great to have a mode where
such details and nit-picks are reported but decoding continues. But we don’t
want to just dump them unstructured to
stderr, like some C libraries and
tools do. The hope behind such an optional validation layer approach is to
provide an incentive for conformance, yet stay flexible where compatibility is
- https://github.com/image-rs/image-png/issues/193: Chunk buffer size is inconsistent with std recommendation
- https://github.com/image-rs/image-png/issues/254: Don’t verify adler32 checksum by default
- https://github.com/image-rs/jpeg-decoder/issues/90: Decoding Twice Error
- https://github.com/image-rs/jpeg-decoder/issues/169: Corrupted JPEG does not result in visible error code
- https://github.com/image-rs/image/issues/1235: error on decoding a damaged-but-readable JPEG file
image library currently does not expose a consistent interface to
supplementary information such as color spaces, comments, copyright,
orientation, and other parts of EXIF. This starts at concrete treatment of
extension chunks in the decoder libraries and goes all the way to design work
for integrating those in the
ImageDecoder trait and the
- https://github.com/image-rs/image-gif/issues/40: Allow reading loop control extension (so that it can be written again)
- https://github.com/image-rs/image-gif/issues/79: The treatment of extensions
- https://github.com/image-rs/image-png/issues/116: Support for encoding metadata chunks
- https://github.com/image-rs/image-png/issues/221: Treatment of unknown chunks
- https://github.com/image-rs/image/issues/258: Need more portable image coordinates.
- https://github.com/image-rs/image/issues/1371: Orientation field not found after load image and save
Limits during decoding
Loading an image from a remote server, only to find out that it decompresses to
several gigabytes of memory and totally grinds your system to a halt/crashes
your program isn’t great. We’d like to ensure that the decoders all have memory
and/or runtime limits that they check and abide to. This also helps with
fuzzing as it in turn allows controlling for such use explicitly. Note that most
of the core libraries (png, gif, tiff) all have their own form of limits but
jpeg-decoder does not. Also, there is no common interface to control
image and to restrict decoding to such parsers that can enforce them.
- https://github.com/image-rs/image/issues/938: Library wide memory limits
- https://github.com/image-rs/image/issues/1052: Large CPU and memory consumption on decoding a crafted GIF file
- https://github.com/image-rs/image/issues/1139: Excessively slow parsing of certain files
- https://github.com/image-rs/jpeg-decoder/issues/133: Introduce limits: Excessive memory allocation for small inputs
- https://github.com/image-rs/jpeg-decoder/issues/134: Slow parsing for small inputs
Code style and Documentation
Have you ever been puzzled by an interface, only to find the correct use some hours later? Answers to these question rarely end up as issues or, even rarer, as PRs. However, consider there is no more understanding of your struggle than yourself. It’s really easy to slip into a mindset where you accept bad ergonomics as quirks and in many cases a single additional sentence of documentation would help tremendously.
There is no dedicated issue tracking for most of these issues (for reasons stated above) but there are a couple:
- https://github.com/image-rs/image-png/issues/203: Document BitDepth::Sixteen encoding
- https://github.com/image-rs/image/issues/598: What is the meaning of conversion from RGB to luma?
- https://github.com/image-rs/image/issues/788: Mention the origin of the coordinate system in README/Documentation
- https://github.com/image-rs/image/issues/1182: Plan for improving the documentation structure
Please note, though, that purely robotic changes are not the goal. Such approaches quickly expand to thousands of changed lines which is hard to review. However, if your editor annoys you greatly with yellow squiggly lines below the code you want to edit then improving the formatting of a single file or chunk of code would be a welcome addition to any otherwise small PR.
Lastly, raising a bit of awareness of an experimental image buffer library in
the hopes of gathering a few new eyes, use cases, and contributors. The
ImageBuffer is showing a few cracks in its design but replacing it directly
is also not quite feasible. It has chosen one layout and encodes this within
its own type. Both of these aspects have proven to be too inflexible.
Vec<T> for representing pixels is not very efficient if one wants to
support operations that change the sample type. This is due to the fact that it
relies on the standard libraries use of allocators and the memory allocated for
a vector is tied to the exact layout of the sample type. There is a concept
library to work around this restriction by storing everything as a highly
aligned byte buffer, which makes it unnecessary to track the sample type as a
type parameter to the buffer type itself. It’s a work in progress found here:
The flexible layout makes it seem like a reasonable goal to create integration
with the various texture layouts found in graphics crates such as
gtk, the Linux DRM module, and
image, and enable some
interoperability between them.
A call for action
The progress report hopefully inspired as much sense of progress as I had
writing this post, and conveyed the vision of where we want to go. You can help
us get there! No matter if you simply want to learn about image formats, if you
want to contribute to the ecosystem, if you your employer needs a specific
feature, or if you’re here for other reasons entirely: Grab an issue,
investigate, open a PR, and we’ll help you get it upstream. Let’s make 2021 at
least as successful for
image-rs as last year.