Implicit Performance Improvements#

This notebook illustrates how the performance is automatically improved when rendering images/labels and what to consider when plotting multiscale-images/-labels. A heuristic approach is used to achieve fast rendering times, however, there are multiple ways to change the default behavior.

The example dataset can be downloaded here.

import warnings

import matplotlib.pyplot as plt
import spatialdata as sd
import spatialdata_plot

warnings.filterwarnings("ignore")
sdata = sd.read_zarr("./visium_associated_xenium_io.zarr")
sdata
SpatialData object with:
├── Images
│     ├── 'CytAssist_FFPE_Human_Breast_Cancer_full_image': MultiscaleSpatialImage[cyx] (3, 21571, 19505), (3, 10785, 9752), (3, 5392, 4876), (3, 2696, 2438), (3, 1348, 1219)
│     ├── 'CytAssist_FFPE_Human_Breast_Cancer_hires_image': SpatialImage[cyx] (3, 2000, 1809)
│     ├── 'CytAssist_FFPE_Human_Breast_Cancer_lowres_image': SpatialImage[cyx] (3, 600, 543)
│     └── 'multicale_copy': MultiscaleSpatialImage[cyx] (3, 21571, 19505), (3, 10785, 9752), (3, 5392, 4876), (3, 2696, 2438), (3, 1348, 1219)
├── Shapes
│     └── 'CytAssist_FFPE_Human_Breast_Cancer': GeoDataFrame shape: (4992, 2) (2D shapes)
└── Table
      └── AnnData object with n_obs × n_vars = 4992 × 18085
    obs: 'in_tissue', 'array_row', 'array_col', 'spot_id', 'region'
    var: 'gene_ids', 'feature_types', 'genome'
    uns: 'spatial', 'spatialdata_attrs'
    obsm: 'spatial': AnnData (4992, 18085)
with coordinate systems:
▸ 'downscaled_hires', with elements:
        CytAssist_FFPE_Human_Breast_Cancer_hires_image (Images), CytAssist_FFPE_Human_Breast_Cancer (Shapes)
▸ 'downscaled_lowres', with elements:
        CytAssist_FFPE_Human_Breast_Cancer_lowres_image (Images), CytAssist_FFPE_Human_Breast_Cancer (Shapes)
▸ 'global', with elements:
        CytAssist_FFPE_Human_Breast_Cancer_full_image (Images), multicale_copy (Images), CytAssist_FFPE_Human_Breast_Cancer (Shapes)

1 Single-Scale Images#

A single-scale image is a simple, single image with values in a grid. On the other hand, a SpatialData object can include multi-scale images that are composed of multiple single-scale images at different scales. All of those single-scale images show the same picture, just at different resolutions (the underlying grids differ). Therefore, one single-scale image can include more details when plotted compared to another single-scale image belonging to the same multi-scale image while having a larger size and higher rendering times.

An image or label can either be single- or multi-scale and the rendering process differs slightly. For a multi-scale image or label, one of the single scales has to be chosen initially in order to get a renderable image which is then treated like a single-scale image. For a single-scale image, one does not have the option to select a scale, but decreasing the resolution in order to increase the runtime is still possible by downsampling the image.

1.1 Default Behavior#

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_lowres_image").pl.show()
INFO     Dropping coordinate system 'downscaled_hires' since it doesn't have relevant elements.                    
INFO     Dropping coordinate system 'global' since it doesn't have relevant elements.                              
CPU times: total: 969 ms
Wall time: 2.22 s
../../../../_images/6f626c0d4920cc5ae9006bab9f6fff506cbdefcdddda46f79e944a91aba669f8.png

When necessary, the image is rasterized to improve performance. Here, the image is downsampled which leads to a lower resolution. This is especially important when the size of the rendering device is smaller than the size of the image to render. In this case, rendering the full image instead of the rasterized one would not lead to a better result while having longer rendering times. Under the hood, a heuristic using image extent, dpi and size of the rendering device determines the necessity of rasterization.

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_hires_image").pl.show()
# the image is automatically rasterized before rendering
INFO     Dropping coordinate system 'global' since it doesn't have relevant elements.                              
INFO     Dropping coordinate system 'downscaled_lowres' since it doesn't have relevant elements.                   
CPU times: total: 1.02 s
Wall time: 1.16 s
../../../../_images/8720377f733805e86173e7624795784a19906bd24a4797cf4f08d82fbf3b2806.png

1.2 Options#

The user can set image size and dpi with the dpi and figisze parameters of pl.show(). These parameters also have an influence on the rasterization! For example, if an image is automatically rasterized, because its size is larger than that of the rendering device, the rasterization is not performed anymore after the figsize is increased to a value larger than the image size. The reason for this is that dpi and figsize are part of the heuristic that is used to decide if rasterization is necessary.

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_hires_image").pl.show(dpi=150)
# the image is automatically rasterized before rendering
INFO     Dropping coordinate system 'global' since it doesn't have relevant elements.                              
INFO     Dropping coordinate system 'downscaled_lowres' since it doesn't have relevant elements.                   
CPU times: total: 1.48 s
Wall time: 1.55 s
../../../../_images/b4b1359cfcb5906323dc2b21c9e541a9fcb6d444ba3f1e347a400c355201a314.png

One can also alter the dpi and figsize parameters in plt.figure() or plt.subplots() which can be seen in the following.

%%time
fig, axs = plt.subplots(ncols=1, figsize=(2, 2))
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_hires_image").pl.show(ax=axs)
INFO     Dropping coordinate system 'downscaled_lowres' since it doesn't have relevant elements.                   
INFO     Dropping coordinate system 'global' since it doesn't have relevant elements.                              
CPU times: total: 719 ms
Wall time: 730 ms
../../../../_images/682522e1f9082ceed8ed2286d566acd65b895f6dee2364fb585e254d2bafc858.png

In order to disable the rasterization of an image, one can set scale="full" in pl.render_images() or pl.render_labels(). Note, that depending on the image size, this can lead to long rendering times. In this case, the image size is not large enough to lead to a “too long” runtime. See a more drastic effect of disabling the rasterization in part 2 (the multi-scale image contains a larger image).

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_hires_image", scale="full").pl.show()
INFO     Dropping coordinate system 'global' since it doesn't have relevant elements.                              
INFO     Dropping coordinate system 'downscaled_lowres' since it doesn't have relevant elements.                   
CPU times: total: 656 ms
Wall time: 676 ms
../../../../_images/4a75aa55c586cb4f6645ae7de0187a133c43c054ac324e2ad83423fd8950c74f.png

2 Multi-Scale Images#

2.1 Default Behavior#

Per default, the scale that fits the size of the rendering device best is automatically selected. From there on, the selected scale is treated like a single-scale image (e.g. if necessary, a rasterization step is added to speed up the rendering process).

In the example here: scale4 (the scale with the lowest resolution) was automatically selected.

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_full_image").pl.show("global")
# "scale4" is automatically selected
CPU times: total: 781 ms
Wall time: 972 ms
../../../../_images/63fce18f610f4ecd45be0b41b4e19cab1381b2c12d8b70affd0f6b288bd7ee7e.png

2.2 Options#

As always, image size and dpi can be regulated via the dpi and figsize parameters of pl.show(). Those parameters also affect the scale selection!

The example below shows that when choosing a higher dpi, a scale with higher resolution is selected automatically (scale3 in this case). Also, a rasterization step was performed to speed up the performance (just as it would happen with a single-scale image). It is the normal behavior that when the “optimal” scale lies between two existing scales, the one with higher resolution is selected and if necessary, the image is rasterized to speed up the rendering. Else, the resulting resolution could be lower than the “optimal” one.

The same behavior could have been achieved with e.g. dpi=300.

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_full_image").pl.show("global", figsize=(15.0, 15.0))
# "scale3" is automatically selected and the image is automatically rasterized before rendering
CPU times: total: 9min 12s
Wall time: 1min 23s
../../../../_images/8865e0eadd5245a7e3933d4263838926d57e14450ca29fe683c8c2d2896a9b1a.png

The user can also select a specific scale to be rendered using the scale argument of pl.render_images() or pl.render_labels().

Note, that when a specific scale is selected, no rasterization will be performed, unless: dpi or figsize are specified by the user in pl.show(). This re-enables the rasterization if neccessary.

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_full_image", scale="scale3").pl.show("global")
CPU times: total: 1.66 s
Wall time: 1.37 s
../../../../_images/1e46244f5eb6778f12b35dba4d054460e6651d22868315a83be23f7eb86b9cc7.png
%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_full_image", scale="scale3").pl.show("global", dpi=100)
# the image is automatically rasterized before rendering
CPU times: total: 35.7 s
Wall time: 16.2 s
../../../../_images/c6dd7eda815f13e38f1661be6c85473ffadb4f5c7a379d08811d421af957f1da.png

Below, you can see an example where the scale with highest resolution was selected but the overall plot should have a dpi of 100. This leads to a drastic rasterization and a suboptimal result regarding image quality. This shows why it usually doesn’t make much sense to specify scale and dpi/figsize at the same time.

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_full_image", scale="scale0").pl.show("global", dpi=100)
# the image is automatically rasterized before rendering
CPU times: total: 1min
Wall time: 2min 1s
../../../../_images/b083f7b2b2a76437527359a40786e576a322e4e4f183c41f0a175a96c23f1d5a.png

Setting scale="full" in pl.render_images() or pl.render_labels() will lead to the scale with highest resolution being selected and no rasterization being performed. Depending on resolution/image size this can lead to long rendering times! In this case, we are looking at the largest image in the SpatialData object (“scale0”). Depending on the machine, the rendering might also run into a memory error which is why the version with scale="full" is not included here. Instead, scale="scale1" is used here to demonstrate high rendering times for images with high resolution.

%%time
sdata.pl.render_images("CytAssist_FFPE_Human_Breast_Cancer_full_image", scale="scale1").pl.show("global")
# using the second highest resolution
CPU times: total: 2min 12s
Wall time: 2min 25s
../../../../_images/e1a69428591119413833dc2ecedf0569de5577fc074120717fd438d22e7746ef.png