Image Catalog Tool
I thought I would start a dedicated section on my site for the Manifold GIS based Image Catalog Tool.
Introduction
Image catalogs are used in ESRI technology, Manifold GIS and software such as the UMN MapServer. My image catalog tool, while not perfect, does (once you have mastered the tool and built some management practices around its use for your organisation) make the creation and management of image catalogs easier. It also facilitates the easy update of imagery in a catalog as well as the creation of single image mosaics from an image catalog.
The image catalog tool is more than just a tool to insert images into an image catalog. The tool comes with two components (a hyperlink means that there is an introductory video associated with the component or function being discussed).
The Catalog Registry Tool is used to create and manage an image catalog. Via this tool one can define an image catalog’s properties (metadata) and have the directory structure and shapefiles created. The metadata that is defined and managed by the Catalog Registry Tool makes the operation of the actual Image Catalog Tool far easier.
Once a catalog is defined one uses the Image Catalog Tool to add imagery to an image catalog. Adding imagery could entail the creation of new tiles or the updating of existing ones.
The Image Catalog Tool has many strengths over existing management tools.
- It is free (expect for a Manifold GIS license of USD250);
- It enables an image catalog’s image quality to improve over time. It does this because if an existing tile has a pixel resolution that is coarser than any new image being added to the catalog that tile is upgraded to the smaller pixel resolution of the new imagery. In this way an image catalog’s image quality only improves over time and does not degrade. (See discussion on “minimum pixel size” in the Catalog Registry Tool.)
- Single image mosaics can be generated from a set of image catalog tiles.
- Multiple tile storage and “world file referencing” formats are supported.
- A history record for each image that has been added to an image catalog can be created. The image’s spatial extent is saved as a footprint that can be visualised in a GIS.
- A boundary shape defining the actual extent of the data is automatically created after each execution of the tool. This is useful for visualising areas in which tiles exist before zooming into particular areas.
The Image Catalog Tool is based on published science. This science allows us to define a optimum tile size for an image catalog so that disk access times are kept to a minimum improving visualisation response times in internet and interactive GIS use.
Over time I will add videos and documentation on all aspects of the tool until such a time as Manifold.Net incorporate the ideas in the tool into the product.
Catalog Registry Script
Image processing cannot be done without afirst a catalog created with the metadata the Image Cataloging tool requires.
The header for the CatalogRegistry script in the imagecatalog.map tool is:
' @application: CatalogRegistryTool
' @precis : A tool for creating the image catalog that the imageCatalogTool requires.
' @version : 1.2
' @usage : Simply select the CatalogRegistryTool component and run. Follow the Tab boxes to set up
' the required processes and execute.
' @requires : The CatalogRegistry table component to exist in the projects pane.
' @return : Nothing.
' @tobedone : 1. Tab order. 2. Directory nomination via dialog.
' @history : Simon Greener, SpatialDB Advisor - Sep 2005 - Original Coding
' @history : Simon Greener, SpatialDB Advisor - Oct 2005 - Added support for history shapefile
' @history : Simon Greener, SpatialDB Advisor - Jan 2006 - Support for user definable tile prefixes
' @history : Simon Greener, SpatialDB Advisor - Feb 2006 - Support for user definable image folder prefix
' - Support for ESRI World Files added
' - Resample to min pixel size, image compression,
' remove projection XML moved from ImageCatalogTool to Registry
Image Catalog Script
The header from the ImageCatalog script within the imagecatalog.map file is:
' @application: ImageCatalog
' @precis : A general purpose tool for working with ESRI-style image catalogs. Similar to
' Microsoft TerraServer's TerraCutter application.
' @version : 3.0
' @description: ESRI Image Catalogs are a method for organising large numbers of images
' under a single structure that can be used as a single "theme" in a mapping application.
'
' Generally, these Image Catalogs are simple and are based on a DBF (or INFO) table
' whose attributes are:
'
' IMAGE - String - Full path name to the actual image file eg c:\myimages\col234\c234r123.tif
' XMIN - Number - X ordinate of Lower Left (LL) corner of the image in real world coordinate system.
' YMIN - Number - Y ordinate of LL corner of image in real world coordinate system.
' XMAX - Number - X ordinate of Upper Right (UR) corner of the image in real world coordinate system.
' YMAX - Number - Y ordinate of UR corner of image in real world coordinate system.
'
' This attribute file-base approach does have shortcomings. For example, it is not possible to
' "visualise" tile boundaries as vector objects. Nor is it easy to see the "extent"
' of the image catalog.
'
' Another limitation of ESRI Image Catalogs is that they do not require that the individual
' images that it references be non-overlapping. They provide no logical metadata to
' control tile creation or organisation.
'
' Constructing Image Catalogs is tedious and requires specialist software: Manifold
' provides all the required functionality to construct individual images but does not
' have the higher level "framework" or "ease of use" tools for constructing actual
' image catalogs.
'
' This Image Catalog application addresses all these issues as follows:
'
' 1. Individual Image Catalogs are hosted as Drawings and exported to disk as
' shapefiles whose DBF files contain the ESRI attributes.
' The approach taken here is similar to UNC's MapServer: image catalogs constructed
' using this tool should work within MapServer.
' 2. An additional shapefile - called the boundary shapefile - is maintained which only
' shows the outer extent of the tiles in the image catalog: this is useful in mapping
' applications where one shows the boundary at small scales and only shows the image
' tiles themselves at larger scales.
' 3. Any single image catalog is defined as a logical grid (like a matrix) over a
' particular piece of geographic space. Each tile or cell in the grid has a predefined
' size in X and Y. The logical grid is "sparse" in that cells are not created unless
' a tile exists in that logcal.
' 4. The size information is used to construct a logical address for each cell as a col/row address.
' The images resulting from each each tile are organised on disk into a hierarchical
' folder structure in which folders are created for each logical column and all row tiles
' in that column are stored in that folder.
' This is done to minimise directory search problems which result in most operating systems
' when the number of files in a directory become too large. This is still fairly coarse and
' could be refined so that the rows were themselves organised perhaps into Base16,32 or 64 buckets.
' Another improvement would be for the images to be organised by a Peano, Morton or
' some other space curve key (this would help in DBMS based image catalogs by creating a single
' searchable primary key column).
' 5. An image catalog has attached projection information.
' 6. An image catalog has a defined minimum precision of the pixels in each tile's image in X and Y.
' New images to be inserted into a catalog can have better precision but images with worse
' precision are resampled to that pixel size IF the user requests. Otherwise they are ignored.
' 7. Image catalog metadata is managed by a CatalogRegistry table (stored on disk as a DBF file).
'
' To minimise the number of disk files all images are stored on disk as GeoTiffs. All
' Manifold XML files are removed after each image has been exported to disk. The XML
' is superfluous as the projection information is a fundamental part of the image catalog's
' metadata.
'
' New images to be inserted into a catalog are processed as follows:
'
' 1. The logical tiles that overlay the image are constructed in the projected space
' of the image catalog.
' 2. Each tile is then processed (via 2 nested for loops).
' 3. Image creation:
' A. If an image already exists in the original image catalog it is loaded.
' B. If an image exists in the output "changes" directory (see later) it is
' loaded in preference to A.
' C. If neither A nor B apply a new image component is created.
' 4. A Query is constructed to select those pixels in the source image that
' pertain to the new image (with sufficient overlap).
' 5. The selected pixels (if non the tile is skipped) are then copied and pasted
' to the image component representing the tile. Note: missing pixels will not
' over-write existing pixels. If the selection is of one colour it is skipped if the
' user ticks the appropriate box.
' 6. The image is written to disk and an entry placed in the image catalog's drawing
' if a new tile had to be created.
'
' When all tiles and images have been processed the image catalog drawing is written
' to disk and any related Boundary shapefile updated via an import, update UnionALL query,
' and export process.
' @usage : Simply select the ImageCatalogTool component and run. Follow the Tab boxes to set up
' the required processes and execute.
' @requires : The following options are required when running this script:
' * Fill new pixels with current colour - Off
' * Import data to separate folders - Off
' * Import empty columns in tables - On
' * Confirm expanding images or surfaces when pasting - Off
' @return : Nothing.
' @tobedone : Mosaic a selection of tiles in an Image Catalog.
' @history : Simon Greener - 2003 - Converted from ArcInfo AML and Avenue.
' Simon Greener - 2004 - Numerous fixes.
' Simon Greener - Jan 2005 - Totally re-wrote using VBScript Classes. Also took advantage of new Query capabilities in 6.00 SP1.
' Simon Greener - Apr 2005 - Fixed bugs and improved performance using new 6.5 features.
' Simon Greener - Sep 2005 - Added in support for CSPreset and Allows for CatalogRegistry component to already exist.
' - Support for Logos in folder not being detected as images to tile
' @History : Simon Greener - Oct 2005 - Added support for history shapefile
' @History : Simon Greener - Jan 2006 - Support for image catalogs being NON-updateable if its CatalogRegistry.OutImgFldr is empty
' - Modified handling of image coordinate system parameters (bug fix).
' - Added in "Remove Projection (XML) capability"
' @history : Simon Greener - Feb 2006 - Support for user definable image folder prefixes
' - Support for CatalogRegistry based RemoveXML, Resample, ESRIWorld, Image Format, Compressions and Options
' - Support for resampling existing tiles if pixelsize > desirable pixel size
' - Added in Mosaicing from Catalog and Components
' - Removed create catalog records only and tile from catalog records options.
' - Integrated Jamie Keene's work wherein a selection that contains a single colour is ignored (using flag)
Image processing cannot be done without afirst a catalog created with the metadata the Image Cataloging tool requires.
The header for the CatalogRegistry script in the imagecatalog.map tool is:
' @application: CatalogRegistryTool ' @precis : A tool for creating the image catalog that the imageCatalogTool requires. ' @version : 1.2 ' @usage : Simply select the CatalogRegistryTool component and run. Follow the Tab boxes to set up ' the required processes and execute. ' @requires : The CatalogRegistry table component to exist in the projects pane. ' @return : Nothing. ' @tobedone : 1. Tab order. 2. Directory nomination via dialog. ' @history : Simon Greener, SpatialDB Advisor - Sep 2005 - Original Coding ' @history : Simon Greener, SpatialDB Advisor - Oct 2005 - Added support for history shapefile ' @history : Simon Greener, SpatialDB Advisor - Jan 2006 - Support for user definable tile prefixes ' @history : Simon Greener, SpatialDB Advisor - Feb 2006 - Support for user definable image folder prefix ' - Support for ESRI World Files added ' - Resample to min pixel size, image compression, ' remove projection XML moved from ImageCatalogTool to Registry
Image Catalog Script
The header from the ImageCatalog script within the imagecatalog.map file is:
' @application: ImageCatalog ' @precis : A general purpose tool for working with ESRI-style image catalogs. Similar to ' Microsoft TerraServer's TerraCutter application. ' @version : 3.0 ' @description: ESRI Image Catalogs are a method for organising large numbers of images ' under a single structure that can be used as a single "theme" in a mapping application. ' ' Generally, these Image Catalogs are simple and are based on a DBF (or INFO) table ' whose attributes are: ' ' IMAGE - String - Full path name to the actual image file eg c:\myimages\col234\c234r123.tif ' XMIN - Number - X ordinate of Lower Left (LL) corner of the image in real world coordinate system. ' YMIN - Number - Y ordinate of LL corner of image in real world coordinate system. ' XMAX - Number - X ordinate of Upper Right (UR) corner of the image in real world coordinate system. ' YMAX - Number - Y ordinate of UR corner of image in real world coordinate system. ' ' This attribute file-base approach does have shortcomings. For example, it is not possible to ' "visualise" tile boundaries as vector objects. Nor is it easy to see the "extent" ' of the image catalog. ' ' Another limitation of ESRI Image Catalogs is that they do not require that the individual ' images that it references be non-overlapping. They provide no logical metadata to ' control tile creation or organisation. ' ' Constructing Image Catalogs is tedious and requires specialist software: Manifold ' provides all the required functionality to construct individual images but does not ' have the higher level "framework" or "ease of use" tools for constructing actual ' image catalogs. ' ' This Image Catalog application addresses all these issues as follows: ' ' 1. Individual Image Catalogs are hosted as Drawings and exported to disk as ' shapefiles whose DBF files contain the ESRI attributes. ' The approach taken here is similar to UNC's MapServer: image catalogs constructed ' using this tool should work within MapServer. ' 2. An additional shapefile - called the boundary shapefile - is maintained which only ' shows the outer extent of the tiles in the image catalog: this is useful in mapping ' applications where one shows the boundary at small scales and only shows the image ' tiles themselves at larger scales. ' 3. Any single image catalog is defined as a logical grid (like a matrix) over a ' particular piece of geographic space. Each tile or cell in the grid has a predefined ' size in X and Y. The logical grid is "sparse" in that cells are not created unless ' a tile exists in that logcal. ' 4. The size information is used to construct a logical address for each cell as a col/row address. ' The images resulting from each each tile are organised on disk into a hierarchical ' folder structure in which folders are created for each logical column and all row tiles ' in that column are stored in that folder. ' This is done to minimise directory search problems which result in most operating systems ' when the number of files in a directory become too large. This is still fairly coarse and ' could be refined so that the rows were themselves organised perhaps into Base16,32 or 64 buckets. ' Another improvement would be for the images to be organised by a Peano, Morton or ' some other space curve key (this would help in DBMS based image catalogs by creating a single ' searchable primary key column). ' 5. An image catalog has attached projection information. ' 6. An image catalog has a defined minimum precision of the pixels in each tile's image in X and Y. ' New images to be inserted into a catalog can have better precision but images with worse ' precision are resampled to that pixel size IF the user requests. Otherwise they are ignored. ' 7. Image catalog metadata is managed by a CatalogRegistry table (stored on disk as a DBF file). ' ' To minimise the number of disk files all images are stored on disk as GeoTiffs. All ' Manifold XML files are removed after each image has been exported to disk. The XML ' is superfluous as the projection information is a fundamental part of the image catalog's ' metadata. ' ' New images to be inserted into a catalog are processed as follows: ' ' 1. The logical tiles that overlay the image are constructed in the projected space ' of the image catalog. ' 2. Each tile is then processed (via 2 nested for loops). ' 3. Image creation: ' A. If an image already exists in the original image catalog it is loaded. ' B. If an image exists in the output "changes" directory (see later) it is ' loaded in preference to A. ' C. If neither A nor B apply a new image component is created. ' 4. A Query is constructed to select those pixels in the source image that ' pertain to the new image (with sufficient overlap). ' 5. The selected pixels (if non the tile is skipped) are then copied and pasted ' to the image component representing the tile. Note: missing pixels will not ' over-write existing pixels. If the selection is of one colour it is skipped if the ' user ticks the appropriate box. ' 6. The image is written to disk and an entry placed in the image catalog's drawing ' if a new tile had to be created. ' ' When all tiles and images have been processed the image catalog drawing is written ' to disk and any related Boundary shapefile updated via an import, update UnionALL query, ' and export process. ' @usage : Simply select the ImageCatalogTool component and run. Follow the Tab boxes to set up ' the required processes and execute. ' @requires : The following options are required when running this script: ' * Fill new pixels with current colour - Off ' * Import data to separate folders - Off ' * Import empty columns in tables - On ' * Confirm expanding images or surfaces when pasting - Off ' @return : Nothing. ' @tobedone : Mosaic a selection of tiles in an Image Catalog. ' @history : Simon Greener - 2003 - Converted from ArcInfo AML and Avenue. ' Simon Greener - 2004 - Numerous fixes. ' Simon Greener - Jan 2005 - Totally re-wrote using VBScript Classes. Also took advantage of new Query capabilities in 6.00 SP1. ' Simon Greener - Apr 2005 - Fixed bugs and improved performance using new 6.5 features. ' Simon Greener - Sep 2005 - Added in support for CSPreset and Allows for CatalogRegistry component to already exist. ' - Support for Logos in folder not being detected as images to tile ' @History : Simon Greener - Oct 2005 - Added support for history shapefile ' @History : Simon Greener - Jan 2006 - Support for image catalogs being NON-updateable if its CatalogRegistry.OutImgFldr is empty ' - Modified handling of image coordinate system parameters (bug fix). ' - Added in "Remove Projection (XML) capability" ' @history : Simon Greener - Feb 2006 - Support for user definable image folder prefixes ' - Support for CatalogRegistry based RemoveXML, Resample, ESRIWorld, Image Format, Compressions and Options ' - Support for resampling existing tiles if pixelsize > desirable pixel size ' - Added in Mosaicing from Catalog and Components ' - Removed create catalog records only and tile from catalog records options. ' - Integrated Jamie Keene's work wherein a selection that contains a single colour is ignored (using flag)