Mosaicing

The Mosaic class allows high speed mosaicing using normalised cross correlation to detect shifts between image frames, and either dead-leaf or alpha-blended insertion of images into a mosaic. The easiest way to use this functionality is to create an instance of Mosaic class and then use Mosaic.add(img) to sequentially register and add image img to the mosaic, and Mosaic.getMosaic() to get the latest mosaic image. Both img and the mosaic are 2D (monochrome) or 3D (colour) numpy arrays.

An example is provided on Github.

Getting Started

Instantiate an object of the Mosaic class using default options and with a mosaic image size of 1000x1000:

from pybundle import Mosaic
mMosaic = Mosaic(1000)

Add an image img to the mosaic:

mMosaic.add(img)

Request the latest mosaic image:

mosaicImage = mMosaic.getMosaic()

The mosaicImage will be a 2D numpy array if img is 2D and a 3D numpy array if img is 3D, in which case the third axis represents the colour channels.

Methods

class pybundle.Mosaic(mosaicSize, **kwargs)

The Mosaic class is used for producing mosaics from a sequence of images. After instantiating a Mosaic object, use add() to add in images and get_mosaic to obtain the current mosaic image.

Parameters:

mosaicSize – int, square size of mosaic image

Keyword Arguments:

  • resize – int, images will be resized to a square this size, default is None meaning no resize.

  • templateSize – int, size of square to extract to use as template for shift detection. Default is 1/4 image size.

  • refSize – int, size of square to extract to use as image to compare template to for shift detection. Default is 1/2 image size.

  • cropSize – int, input images are cropped to a circle of this diameter before insertion. (default is 0.9 x size of first image added)

  • imageType – str, data type for mosaic, default is the same as first image added

  • blend – boolean, if True (default), images will be added blended, otherwise they are added dead-leaf

  • blendDist – int, distance in pixels from edge of inserted image to blend with mosaic, default is 40

  • minDistforAdd – int, minimum distance moved before an image will be added to the mosaic, default is 25

  • initialX – int, starting positon of mosaic, default is centre

  • initialY – int, starting positon of mosaic, default is centre

  • boundaryMethod – method to deal with reaching edge: CROP, SCROLL or EXPAND, default is CROP

  • expandStep – int, amount to expand by if EXPAND boundaryMethod is used

  • resetThresh – float, mosaic will reset if correlation peak is below this, default is None (ignore)

  • resetIntensity – float, mosaic will reset if mean image value is below this value, default is None (ignore)

  • resetSharpness – float, mosaic will reset if image sharpness (mean of gradient) drops below this value, default is None (ignore)

add(img)

Add image to current mosaic.

Parameters:

img – image as 2D/3D numpy array

get_mosaic()

Returns current mosaic image as 2D/3D numpy array

reset()

Call to reset mosaic, clearing image. The mosaic will only be fully reset once a new image is added. Note that parameters that were already initialised will not be reset.

Usage Notes

The only required argument is the size of the mosaic image. By default images will be added blended, there will be no resize of the input image, no checking of input image quality and if the mosaic reaches the edge of the image it will simple run off the the edge.

Usually it is beneficial to resize the input images to prevent the need for a very large mosaic image, e.g.:

mMosaic = Mosaic(1000, resize = 250)

The reset methods (resetThresh, resetIntensity and resetSharpness) are normally required when used with a handheld probe to handle instances where tissue contact is lost or the probe is moved too quickly. For optical sectioning endomicroscope, a combination of correlation based thresholding (resetThresh) and intensity based thresholding (resetIntensity) works well. For non-sectioning endomicroscopes, moving out of focus does not sufficiently reduce either, and so it may be necessary to use sharpness thresholding (resetSharpness) as well. The best values to use must be determined empirically and will depend on pre-processing steps.

For slow moving probes, minDistForAdd may need to be adjusted particularly when using blending to prevent undesirable effects of the same image being blended with itself.

Low Level Functions

The private member functions of the Mosaic class are listed below for custom use:

class pybundle.Mosaic(mosaicSize, **kwargs)

The Mosaic class is used for producing mosaics from a sequence of images. After instantiating a Mosaic object, use add() to add in images and get_mosaic to obtain the current mosaic image.

Parameters:

mosaicSize – int, square size of mosaic image

Keyword Arguments:

  • resize – int, images will be resized to a square this size, default is None meaning no resize.

  • templateSize – int, size of square to extract to use as template for shift detection. Default is 1/4 image size.

  • refSize – int, size of square to extract to use as image to compare template to for shift detection. Default is 1/2 image size.

  • cropSize – int, input images are cropped to a circle of this diameter before insertion. (default is 0.9 x size of first image added)

  • imageType – str, data type for mosaic, default is the same as first image added

  • blend – boolean, if True (default), images will be added blended, otherwise they are added dead-leaf

  • blendDist – int, distance in pixels from edge of inserted image to blend with mosaic, default is 40

  • minDistforAdd – int, minimum distance moved before an image will be added to the mosaic, default is 25

  • initialX – int, starting positon of mosaic, default is centre

  • initialY – int, starting positon of mosaic, default is centre

  • boundaryMethod – method to deal with reaching edge: CROP, SCROLL or EXPAND, default is CROP

  • expandStep – int, amount to expand by if EXPAND boundaryMethod is used

  • resetThresh – float, mosaic will reset if correlation peak is below this, default is None (ignore)

  • resetIntensity – float, mosaic will reset if mean image value is below this value, default is None (ignore)

  • resetSharpness – float, mosaic will reset if image sharpness (mean of gradient) drops below this value, default is None (ignore)

__cosine_window(circleSize, circleSmooth)

Produce a circular cosine window mask on grid of imgSize * imgSize. Mask is 0 for radius > circleSize and 1 for radius < (circleSize - circleSmooth). The intermediate region is a smooth cosine function.

Returns mask as 2D numpy array.

Parameters:

  • imgSize – int, size of square mask to generate

  • circleSize – int, radius of mask (mask pixels outside here are 0)

  • circleSmooth – int, size of smoothing region at the inside edge of the circle. (mask pixels with a radius less than this are 1)

__expand_mosaic(distance, direction, currentX, currentY)

Increase size of mosaic image by ‘distance’ in direction ‘direction’. Supply currentX and currentY position so that these can be modified to be correct for new mosaic size.

Returns tuple of (newMosaic, width, height, newX, newY), where newMosaic is the larger mosaic image as 2D numpy array, width is the x-size of the new mosaic, height is the y-size of the new mosaic, newX is the x position of the last image insertion in the new mosaic, newY is the y position of the last image insertion in the new mosaic.

Parameters:

  • mosaic – input mosaic image as 2D numpy array

  • distance – pixels to expand by

  • direction – side to expand, one of Mosaic.Top, Mosaic.Bottom, Mosaic.Left or Mosaic.Right

  • currentX – x position of last image insertion into mosaic

  • currentY – y position of last image insertion into mosaic

__find_shift(img2, templateSize, refSize)

Calculates how far img2 has shifted relative to img1 using normalised cross correlation.

Returns tuple of (shift, max_val) where shift is a tuple of (x_shift, y_shift) and max_val is the normalised cross correlation peak value. Returns None if the shift cannot be calculated.

Parameters:

  • img1 – reference image as 2D/3D numpy array

  • img2 – template image as 2D/3D numpy array

  • templateSize – int, a square of this size is extracted from img as the template

  • refSize – int, a square of this size is extracted from refSize as the template. Must be bigger than templateSize and the maximum shift detectable is (refSize - templateSize)/2

__initialise_mosaic(img)

Choose sensible values for non-specified parameters.

Parameters:

img – input image to used to choose sensible parameters for mosaicing, 2D/3D numpy array

__insert_into_mosaic(img, mask, position)

Dead leaf insertion of image into a mosaic at specified position. Only pixels for which mask == 1 are copied.

Parameters:

  • mosaic – current mosaic image, 2D/3D numpy array

  • img – img to insert, 2D/3D numpy array

  • mask – 2D numpy array with values of 1 for pixels to be copied and 0 for pixels not to be copied. Must be same size as img.

  • position – position of insertion as tuple of (x,y). This is the pixel the centre of the image will be at.

__insert_into_mosaic_blended(img, mask, blendMask, cropSize, blendDist, position)

Insertion of image into a mosaic with cosine window blending. Only pixels from image for which mask == 1 are copied. Pixels within blendDist of edge of mosaic (i.e. radius of cropSize/2) are blended with existing mosaic pixel values

Parameters:

  • mosaic – current mosaic image, 2D/3D numpy array

  • img – img to insert, 2D/3D numpy array

  • mask – 2D numpy array with values of 1 for pixels to be copied and 0 for pixels not to be copied. Must be same size as img.

  • blendMask – the cosine window blending mask with weighted pixel values. If passed empty [] this will be created

  • cropSize – size of input image.

  • blendDist – number which controls the sptial extent of the blending

  • position – position of insertion as tuple of (x,y). This is the pixel the centre of the image will be at.

__is_outside_mosaic(img, position)

Checks if position of image to insert into mosaic will result in part of inserted image being outside of mosaic. Returns tuple of boolean (true if outside), side it leaves (using consts defined above) and distance is has strayed over the edge. e.g. (True, Mosaic.Top, 20).

Returns tuple of (ouside, side, distance), where outside is True if part of image, side is (one of the) side(s) it has strayed out of, one of Mosaic.Top, Mosaic.Bottom, Mosaic.Left or Mosaic.Right or -1 if outside == False, distance is distance it has strayed outside mosaic in the direction specified in size (0 if outside == False).

Parameters:

  • mosaic – mosaic image (strictly can be any numpy array the same size as the mosaic)

  • img – image to be inserted as 2D numpy array

  • position – position of insertion as tuple of (x,y). This is the pixel the centre of the image will be at.

__scroll_mosaic(distance, direction, currentX, currentY)

Scroll mosaic to allow mosaicing to continue past edge of mosaic. Pixel values will be lost. Supply currentX and currentY position so that these can be modified to be correct for new mosaic size.

Return: tuple of (newMosaic, width, height, newX, newY), where newMosaic is the larger mosaic image as 2D numpy array, width is the x-size of the new mosaic, height is the y-size of the new mosaic, newX is the x position of the last image insertion in the new mosaic, newY is the y position of the last image insertion in the new mosaic.

Parameters:

  • mosaic – input mosaic image as 2D numpy array

  • distance – pixels to expand by

  • direction – side to expand, one of Mosaic.Top, Mosaic.Bottom, Mosaic.Left or Mosaic.Right

  • currentX – x position of last image insertion into mosaic

  • currentY – y position of last image insertion into mosaic

Example

An example is provided in “examples\mosaicing_example.py”