The last year was mainly one of stability. Working up to 0.23 we had fixed many outstanding stability risks and it continues to be one of the most long-serving major versions in a while. In this post you find an overview on the massive progress made with it, and outstanding issues that you could contribute towards in 2021.

We also have a logo now:

Developments in 2020

Part of the stability work was the ability to upgrade the crates with decoder implementations such as gif without breaking the API of the image crate. This allowed you to silently benefit from the improved gif decoder as detailed above and other similar fixes across other decoders and it will also enable similar updates for png soon.

This is directly reflected in dependencies and downloads. While previously there was a long tail, the dependencies and downloads are now dominantly for recent versions and thus quicker, using less memory, more correct and with less panics. (If you’re still using 0.18 or 0.22, you should really consider updating). We’ve also more than doubled our total downloads over the last year! Around 1_000 (one thousand!) crates now depend on at least one of the crates under our organization. That’s mind boggling.

Despite the API stability, we’ve made a ton of progress!

Formats

The list of supported formats has expanded greatly. The formats farbfeld, tga, avif and several DDS subtypes are now recognized or can be written. The existing formats also cover more of the specification.

Let’s start with the basics. The png library now supports decoding of individual APNG frames. It also now treats all allowed pixel formats properly, including sixteen bit depth. This all makes it pass the extended pngsuite with flying colors. It also gained the ability to encode indexed data directly.

The tiff crate has also seen a lot of attention. Throughout the year various contributors added BigTIFF and basic GeoTIFF support, have expanded the supported sample formats to 32-bit and 64-bit color depth, and overall increased the amount of recognized tags. It also decodes embedded modern jpeg image data. Nevertheless, one particularly sore point remains: tiff does not yet support Tile Offsets which would be extremely valuable for access to large data from medical and geodetic data. We’d be delighted to see contributions for this feature.

The jpeg-decoder crate now correctly decodes all progressive images. It now collects ICC and Exif data and makes them available in their raw form. The implementation exposing these in a general interface within image is expected to commence sometime soon.

We’ve tested the new implementations on web data dumps of hundreds of thousands of images from the web—totalling terabytes of data— and decoded the overwhelming majority of them successfully.

Performance

The gif crate joins png in being on par with the relevant C implementations in terms of performance! This was made possible by a combination of improvements. Firstly, the restructured decoder interface make it possible to write directly into the output buffer which avoids a few copies. Secondly, the decompression libraries have been upgraded.

These gains are thanks to a new LZW encoding/decoding library, called weezl. It significantly improves on lzw crate in performance while staying 100% safe, and also had an optional no_std mode. There is integration with both std and async IO-interfaces, behind separate features. weezl, written by the author of this post, is also maintained under the image-rs organization.

Since most of the time in decoding GIF files is spent in LZW decoding, the gains from LZW have naturally translated to gif crate. TIFF files encoded with LZW compression also benefit from this change. The total decoding throughput quadrupled in some instances! The latest versions of gif, tiff and image all take advantage of this new library.

The jpeg-decoder crate has also been sped up significantly. Idiomatic iterator based loops of Rust allow the LLVM compiler to aggressively use auto vectorization. That is, instead of performing a transformation on single elements at a time this optimization technique automatically replaces code operating on elements sequentially (in this case, pixels) with special SIMD instructions that perform them on multiple elements at a time, with large speedups of 400% or more. While not yet on-par with handwritten assembly this can get very close with the right code structure.

Additionally, a number of needless bounds checks have been identified. It was possible to remove them with an iterator-based approach, which improved performance without sacrificing safety. Together, the whole decoding process takes 20%-30% less processor time. Finally, the decoder now makes better use of available parallelism. It now spawns a worker thread for each individual component which substantially reduces wall clock time on multi core systems.

One significant outstanding opportunity at the moment is the suboptimal implementation of the Huffman decoding process. It takes up a disproportionate fraction of the overall time and blocks all worker threads. You can find more information in jpeg-decoder#155 and the Huffman source code.

Downstream Users

There have also been a number of notable new or improved dependent crates. What sets this year apart from previous years is the impression that an increasing fractions of these are end-user applications. And Rust is used a tool for delivering to larger audience than the Rust ecosystem itself, with speed, reliability and productivity.

  • emulsion: A fast and minimalistic image viewer.
  • czkawka: Multi functional app to find duplicates, empty folders, similar images etc.
  • img_hash: A Rust library for calculating perceptual hash values of images.
  • nannou: A Creative Coding Framework for Rust.
  • gst-plugins-rs: Rust gstreamer plugins with various image and video options that use image-rs crates.

Open ideas for 2021

There are still many outstanding issues and unimplemented improvements. From the point of view of a maintainer, some of the most pressing ones are listed in the next sections, sorted roughly by topic.

Fallbacks and Error logs

The decoders are strict, in many ways. There exist a few fallbacks here and there, but we still occasionally encounter a technically invalid image that is successfully decoder by other tools. (Some formats take more liberties than others). The source of this trouble is that being too lenient in decoders leads to an ossification of formats where particular non-standard behavior by the most popular encoders is de-facto standardized without appearing in any specification. For many older formats that ship has, however, long sailed. We should thus accept the popular interpretation (e.g. ImageMagick) of such extensions.

The issue goes a bit farther than this. It would be great to have a mode where such details and nit-picks are reported but decoding continues. But we don’t want to just dump them unstructured to stderr, like some C libraries and tools do. The hope behind such an optional validation layer approach is to provide an incentive for conformance, yet stay flexible where compatibility is prioritized.

Related issues:

Supplementary information

The image library currently does not expose a consistent interface to supplementary information such as color spaces, comments, copyright, orientation, and other parts of EXIF. This starts at concrete treatment of extension chunks in the decoder libraries and goes all the way to design work for integrating those in the ImageDecoder trait and the Reader.

Limits during decoding

Loading an image from a remote server, only to find out that it decompresses to several gigabytes of memory and totally grinds your system to a halt/crashes your program isn’t great. We’d like to ensure that the decoders all have memory and/or runtime limits that they check and abide to. This also helps with fuzzing as it in turn allows controlling for such use explicitly. Note that most of the core libraries (png, gif, tiff) all have their own form of limits but notably jpeg-decoder does not. Also, there is no common interface to control them in image and to restrict decoding to such parsers that can enforce them.

Code style and Documentation

Have you ever been puzzled by an interface, only to find the correct use some hours later? Answers to these question rarely end up as issues or, even rarer, as PRs. However, consider there is no more understanding of your struggle than yourself. It’s really easy to slip into a mindset where you accept bad ergonomics as quirks and in many cases a single additional sentence of documentation would help tremendously.

There is no dedicated issue tracking for most of these issues (for reasons stated above) but there are a couple:

Please note, though, that purely robotic changes are not the goal. Such approaches quickly expand to thousands of changed lines which is hard to review. However, if your editor annoys you greatly with yellow squiggly lines below the code you want to edit then improving the formatting of a single file or chunk of code would be a welcome addition to any otherwise small PR.

Image Buffers

Lastly, raising a bit of awareness of an experimental image buffer library in the hopes of gathering a few new eyes, use cases, and contributors. The ImageBuffer is showing a few cracks in its design but replacing it directly is also not quite feasible. It has chosen one layout and encodes this within its own type. Both of these aspects have proven to be too inflexible.

Using Vec<T> for representing pixels is not very efficient if one wants to support operations that change the sample type. This is due to the fact that it relies on the standard libraries use of allocators and the memory allocated for a vector is tied to the exact layout of the sample type. There is a concept library to work around this restriction by storing everything as a highly aligned byte buffer, which makes it unnecessary to track the sample type as a type parameter to the buffer type itself. It’s a work in progress found here:

https://github.com/image-rs/canvas

The flexible layout makes it seem like a reasonable goal to create integration with the various texture layouts found in graphics crates such as wgpu, ash, gtk, the Linux DRM module, and image, and enable some interoperability between them.

A call for action

The progress report hopefully inspired as much sense of progress as I had writing this post, and conveyed the vision of where we want to go. You can help us get there! No matter if you simply want to learn about image formats, if you want to contribute to the ecosystem, if you your employer needs a specific feature, or if you’re here for other reasons entirely: Grab an issue, investigate, open a PR, and we’ll help you get it upstream. Let’s make 2021 at least as successful for image-rs as last year.