================================= Introduction to Spatial Indexing ================================= Traditional indexing works extremely well for traditional data. Spatial data, however, does not fit the expected model of trafditional data that RDBMSs like to use. Spatial Data Types --------------------- There are three general spatial data types that are typically used to represent geographic reality. We will focus on two dimensional types. The three types are: **point, line, and region**. Furthermore, they are differentiated into two classes: **simple** and **complex**. .. admonition:: Definition: Simple point, line, and region A simple point is a single point in euclidean space: (x,y) A simple line is a connected, one dimensional object in euclidean space with two end points. A simple region is a single, simple, minimal cycle separating the interior of the region from the exterior of the region (A Jordan curve) .. image:: z_static/simplePLR.* :width: 6in .. admonition:: Definition: Complex point, line, and region A complex point, sometimes called a multi-point, is a collection of non-overlapping, simple points A complex line is a collection of disjoint simple lines A complex region is a collection of faces and holes. A face is a simple region, possibly containing holes. A hole fits the definition of a simple region, but the interior of a region lies on the exterior of a hole, and the exterior of a region lies on the interior of a hole. Faces can contain multiple holes, and multiple faces can occur within a hole. Faces and holes can touch along their boundaries at a finite number of discrete points. .. image:: z_static/complexPLR.* :width: 6in In mathematical terms, all data types are considered to be regular open point sets. This has desirable properties for the semantics of operations. All spatial objects have three components: an interior, and exterior, and a boundary. * Points have only a boundary and an extrior. * Lines have a boundary defined as the end points of the line. The interior is all parts of the line not at the end points. The exterior is everything else (the rest of the embedding space) * Regions have a boundary that seperates the interior and exterior of the region. Why are simple data types not sufficient? (think in terms of geographic reality, operations that must be performed, and object oriented conceptualization). Integration of Spatial Types into Databases --------------------------------------------- The mathematical definitions of spatial types are based on point set topology. Types are defined by using infinite point sets, not directly representable in computers. So, we discretize them by storing their boundaries. Furthermore, boundaries are stored as straight line segments. Straight line segments are easy, they have easy geometry, they require no computation to generate, and they are accurate enough. * GPS is accurate to about 5 meters (with a good signal). * 1 meter (approximately within the US and similar lat/lons) is approximately 0.00001 degrees lat/lon. 6 digits of precision fits nicely into a floating point value on computers! * Wars tend to be fought over rivers, not 5 meters of land. (Georgia/Tenessee water wars!) So... how many straight line segments does it take to represent Texas, Louisiana, Arkansa? Should we shoot for compact representations, or more detailed representation. what about portions of boundaries that are straight? Take a look at the TIGER data ( The image below is published by the US. Census Bureau at http://www.census.gov/geo/maps-data/maps/reference-cd113.html ) ... .. image:: z_static/census_congressional_map_tx.* :width: 6in Databases love fixed size records. How do we incorporate spatial data .. admonition:: Concept: Spatial Data In RDBMS RDBMSs like fixed sized records, in fact, many traditional index structures rely on this. Solution: In a table, create a column of a geometry type. The record simply contains a pointer (disk address) of the actual geometry. 1 level of indirection. Counties (Name string(30), stateAbbrv string(2), population int, geometry region) Now, records get to use the existing index structures, but we have the ability to add new index types for the spatial data. Types of Queries for Points ------------------------------ When considering index structures, it make sense to consider the types of queries that will access them. Recall the B+ tree's ability to link adjacent leaf nodes supporting range queries. For point data, there are lots of possible types of queries, here are some common ones: * KNN: *Find the* :math:`k` *nearest neighbors to a query point.* The value of :math:`k` can be 1 to find the nearest neighbor, in which cas it is just a nearest neighbor query. * Range: *Find the points that lie in a query window*. Sometimes called a window query, box query, etc. The user supplies the rectangle defining the window. * Selection: *Find all points equal to a query point* or *find all points within 2 miles of a query point*. Typically used in conjucntion with a predicate or operation (ie., **equal** or **withinRadius**). What are some examples of real world applications that use these query types? Consider ecology, biology, mapping, climatology, anything else you can think of. Grid Indexing ---------------- We typically know something about the data. For example, how much to expect, roughly where it will be, some kinds of bounds. Simplist way to index space just break it up into a uniform grid. .. image:: z_static/uniformGrid.* :width: 6in Computational based. We don't have to store the grid cells. For a universe from 0-10 in X and Y direction, and 10 cells in each dimension, what cell does the point :math:`(5.5,7.5)` lie in? :math:`(cell_x, cell_y)` where :math:`cell_x = 0+ x/((10-0)/10)`. Where do we store the actual points (remember our disk page behavior)? Each grid cell must point to a disk page where the points in that cell are stored. If a page fills up, we simply chain them, much like the leaves of a B+tree. So, we need an index file that translates grid cells to disk pages. .. admonition:: Queries What is the procedure to compute a nearest neighbor query for the point :math:`(5.1, 6.5)`? In the BEST case, how many disk pages do we read? In the worst case, how many disk pages do we read? How many times do we query the index file? Repeat the question for :math:`(8.5, 8.5)`. What is the procedure to compute a range query: "Find all points in the rectangle defined by :math:`(4,4),(6,7)`. Best case disk page reads? worst case disk page reads? What is the procedure to find all points in within a radius of 2 from :math:`(5.1, 6.5)`? Best case disk page reads? worst case disk page reads? One problem for grid files, the disk pages for each cell are not necessarily stored near each other on disk! Advantages: 1. simple 2. can utilize a hash table to make the actual index file smaller 3. If the grid is densly packed, you don't have the problem of empty cells! Disadvantages: 1. sparsely packed grids must indicate that cells are empty somehow, it might mean :math:`O(n^2)` space in the index file. Possibly list in the index file that the cell is empty. Possibly use a hash table approach. 2. Does not scale to regions particularly effectively. 3. How do we store cells consecutively on disk? We can't! disk is a one dimensional structure, we are storing a 2 dimensional structure! Z-Ordering --------------------- Clearly, 2 adjacent grid cells in a grid index cannot always be adjacent on disk (a cell has 8 neighbors!). How do we arrange cells on disk to improve seek time and reduce effective rotational latency? Instead of random cell placement on disk, follow some kind of ordering. Z ordering is a method of mapping 2 (or higher) dimensions to a lower dimensional ordering (typically 1). We want a two dimensional cell structure to map to a 1 dimensional disk addressing scheme such that cell neighbors tend to end up close to each other on disk. Z ordering works by interleaving the bit representations of a cell address: .. image:: z_static/zorder.* :width: 6in If we interleave the bits of each cell address, we end up with a linear ordering of the cells. You can see where the Z ordering gets its name: .. image:: z_static/zorderDrawn.* :width: 6in For example, the cells (0,0), (0,1), (1,0), (1,1) get the following interleaved bit addresses (red bit goes first and are numbers, green bits are **o** and **t**): 0 o 0 o 0 o 0 o: (0,0) = 0 0 o 0 o 0 o 0 t: (0,1) = 1 0 o 0 o 0 o 1 o: (1,0) = 2 0 o 0 o 0 o 1 t: (1,1) = 3 0 o 1 o 0 t 1 o: (2,5) = 38 0 o 1 o 0 t 1 t: (3,5) = 39 .. admonition:: Queries: What is the procedure to compute a nearest neighbor query for the point :math:`(3.5, 4.5)`? In the BEST case, how many disk pages do we read? In the worst case, how many disk pages do we read? How many times do we query the index file? What is the procedure to compute a range query: "Find all points in the rectangle defined by :math:`(4,4),(6,7)`. Best case disk page reads? worst case disk page reads? What is the procedure to find all points in within a radius of 2 from :math:`(4, 6)`? Best case disk page reads? worst case disk page reads? **So... why is the z-ordering superior to the straight grid file? Spatial Proximity** K-d Tree --------------------- The problem with grid based stuctures is that they are static! They do not adapt effectively to the input. In other words, they are highly sensitive to data skew, in which a lot of the data might exist in a small area of the grid. If the data is evenly distributed, grids work well. The k-d tree is an adaptive data structure that will adjust to fit the data. It is a **binary tree** in which every node represents a point in one of **k** dimensions. Non leaf nodes effectively divide a hyperplane into two half-spaces (two parts). Each level in the tree typically cycles among which axis the hyperplane will be perpendicular to. Items that are left of the hyperplane will go into a left subtree, items right of a hyperplane will go in a right subtree. Construction of the tree is as follows: 1. the axis used to generate splitting hyperplanes are cycled repeatedly 2. node points are selected by taking the median of the points being placed in the subtree .. admonition:: Example Lets build a k-d tree with the points: (1,6), (2,3), (3,2) (4,5), (5,4), (6,1). We will start splitting on the x dimension. A split will stop when it enocounters a previosly existing split line. Root node: split on x axis at median of x values = 4: (4,5) left subtree will have: (1,6), (2,3), (3,2) split on y axis at median of y values: 3: (2,3) left subtree will have: (3,2) -- make final split on x axis right subtree will have: (1,6) -- make final split on y axis right subtee will have: (5,4), (6,1) split on y axis of median of y values 4: (5,4 ) left subtree will have: (6,1) -- make final split on x axis .. image:: z_static/kd_empty.* :width: 6in .. image:: z_static/kd.* :width: 6in .. image:: z_static/kd_tree.* :width: 6in Note that the X levels in the tree act like the oprdering of a binary search tree, as do the Y levels. This property allows us to discover the rectangular subdivision of space implied by the tree rather easily. For example, to find the rectangle contianing the point (3.5,3.5), we simply follow the tree like a binary search tree, keeping track of the relevant extrema values: * less than (4,5) on an x split, so the rectangle (l,r,b,t) = (l,4,b,t) * greater than (2,3) on the y split, so (l,4,3,t) * greater than (1,6) on the x split, so (1,4,3,t) Therefore, the contianing rectangle ranges are 1 to 4 in the x direction, and 3 to +infinity in the y direction **Inserting**: Inserting follows the same binary search tree procedure to identify the insertion point. Insertions occur at leaves. **Deleting**: find the node to delete, and then rebuild the tree consisting of all children of that node. OR, do a binary search tree style delete .. admonition:: Properties Using the tree creation algorithm, a K-d tree is balanced (we always choose the median value). Inserts and deletes may cause the tree to become unbalanced. Thus, it is not a self-balancing search tree. Time complexity of building the tree: 1. To find the median value of a particular dimension, we scan all :math:`n` input points. 2. We can then arrange all points to be on the appropriate side of the median point. 3. for each side, we repeat at the next level, again scanning all :math:`n` input points. 4. This process is repeated :math:`lg~n` times because it forms a balanced binary tree. Because $k$ is an input parameter, it affects the number of comparisons to splitting planes. Final time complexity is :math:`O(kn~lg~n)` for building the tree. The basic k-d tree cannot be re-balanced, but adaptive k-d trees do exist. Nearest Neighbor Search __________________________ Basic approach to the nearest neighbor: 1. Find the insertion point if we were to insert the point. Record the parent of that node as the closest. 2. Walk back up the tree, if a node is closer, keep that one. If a closer node could exist on the other side of the splitting plane, check that sub tree. .. admonition:: Example Find the nearest neighbor in the above tree to (4.1, 2) Search the tree to identify node (6,1) with distance = :math:`\sqrt (1.9^2 - 1^2) = 1.6` back up to (5,4) with distance = 1.7 ((6,1) is still closest), :math:`2+1.6= 3.6, 3.6<4`, so no need to search the right child of (5,4) back up to (4,5) with distance = 2.9 ((6,1) is still closes), :math:`4.1-1.6 = 2.5 < 4`, so we need to search the left child of (4,5) recursively search. Find the insert point as a child of (3,2). with distance = 1.05 **closest!** back up to (2,3) with distance = 1.8 ((3,2) is still closest). :math:`3+1.8 = 4.8, 4.8 > 3`, so we need to check right child of (2,3) find (1,6), which is farther than 1.05, so back up the tree already checked both sides of (2,3) already checked both sides of (4,5) Quad Tree ----------------------- Quad trees are another popular alternative to the fixed spatial decomposition scheme of grid files. Quad trees recursively divide space into quadrants until each quadrant contains a maximum number of elements. The maximum number of elements is a decision made before the quad tree is constructed, and will often depend on a disk page size. In essence, the maximum number of elements in a quadrant should be manageable based on some designe constraint. Let :math:`m` be the maximum number of elements grouped together into a leaf node of a quad tree for point data. The basic idea is to begin with the entire embedding space, and see if the are more than :math:`m` elements. If so, then the space is divided into 4 equal quadrants. A root node is created, and leaf nodes, one per quadrant, are attached. Points in each quadrant are recorded in the corresponding leaf node. The procedure is applied recursively to each leaf node. If a leaf node contains fewer then :math:`m` points, the we do not subdivide the node.. .. admonition:: Example Create a quad tree using the following data set.. .. image:: z_static/quad.* :width: 6in Compare the spatial decomposition to that of the uniform grid: .. image:: z_static/uniformGrid.* :width: 6in Avantages: * It is an adaptive spatial decomposition scheme. It adapts to the data, much like a k-d tree. * A node can hold multiple data elements, unlike the basic k-d tree. * The tree structure is readily visible by a visualization of the quadrants. Much more so than a k-d tree. Disadvantages: * It is limited in use as a disk based index structure because the internal nodes are nowhere near the size of a disk page. Schemes can be achieved that pack internal nodes into disk pages, but this is less efficient than a structure that is naturally aligned with disk-page-sized nodes. .. admonition:: Queries What is the procedure to compute a nearest neighbor query for the point :math:`(5.1, 6.5)`? In the BEST case, how many disk pages do we read? In the worst case, how many disk pages do we read? How many times do we query the index file? Repeat the question for :math:`(8.5, 8.5)`. What is the procedure to compute a range query: "Find all points in the rectangle defined by :math:`(4,4),(6,7)`. Best case disk page reads? worst case disk page reads? What is the procedure to find all points in within a radius of 2 from :math:`(5.1, 6.5)`? Best case disk page reads? worst case disk page reads? R-Tree ------------------- Quad trees intuitive, they are adaptable to the data distribution, and they provide simple query constructions, but they do not generalize to a disk based structure very effectively. R-trees are the answer to that drawback of quad trees. The idea of an R-tree is to generalize a B-tree to handle two (or multi-) dimensional data. The first problem we run into is that the problem of dimensionality with spatial data. B-trees rely on a total ordering or integers that corresponds to physical nearness across the dimensionality of the data (one dimensional integers). Points, are two dimensional. We can define a total ordering of points, but two points that are adjacent in ordering are not necessarily physically near to each other. The ordering does not imply physical nearness. .. admonition:: Exercise Define a total ordering over points. Draw three points where two points that adjacent in the ordering are not the nearest points in the embedding space. Do the same with regions. The idea of the R-tree is that physically near objects are grouped together in nodes in the R-tree. Like a B-tree, actual objects are stored only at leafs, and internal nodes provide a search path to the appropriate leaves. R-trees are balanced, have a fixed node size, and have constraints as to the minimum amount of objects allowed in a node. In essence, R-trees are similar in concept to B-trees, but with modifications to handle multi-dimensional data. The main problem with R-trees is making sure that internal nodes don't overlap too much, internal nodes don't contain too much empty space, and that objects that are close together in space are actually stored close together in the tree. An internal tree node will contain minimum bounding rectangles (MBRs). Associated with each MBR is a **single** pointer to a node such that the MBR minimally encloses all spatial objects in that node. (Note the single pointer, as opposed to a data element separating two pointers in a B-tree). When a node is full, it is split, objects are assigned to new nodes, and the MBRs of each node are adjusted. A MBR is then promoted (like a B-tree) to the parent node. A split can propogate all the way to the root, possibly growing the tree by one level. .. admonition:: Example In this example, we will build up the R tree by adding a single regions at a time. Assume an R-tree that holds at most 3 object pointers per node. Adding regions to the root node is straightforward, and behaves much like a B tree. .. image:: z_static/r123.* :width: 4in Adding the fourth region requires a split. We must choose which regions to group together. Ideally regions near each other will be grouped together. .. image:: z_static/r1234.* :width: 4in To insert the 5th region, we must choose which bounding rectangle to expand. Again, we want to use some heuristic, such as the expansion that covers the least additional area. .. image:: z_static/r12345.* :width: 4in Adding the 6th region requires another split, and the root node is now full. We go ahead and add the 7th region since it does not force a split .. image:: z_static/r12346.* :width: 4in Finally, the eight node will force a split, and the promoted bounding rectangle will also force a split, causing a split all the way up to the root .. image:: z_static/r1234678.* :width: 4in An alternative, and better, choice of bounding rectangles when inserting the 8th region. Why is it better? .. image:: z_static/r1234678alt.* :width: 4in Note that algorithms exist to efficiently build an R tree from a set of input regions. Such algortihms can have much better performance than continously inserting regions one by one because they avoid node splitting and tree growing, resulting in fewer disk I/Os. .. admonition:: Queries What is the procedure to compute a nearest neighbor to region 1? In the how many disk pages do we read? Repeat for Region 7. What is the procedure to compute a range query: "Find all points in the rectangle defined by an MBR surrounding regions 7,3, and 5? How many disk page reads? Advantages: * Adapts to distribution of spatial objects. * Aligns well with disk architecture * Handles all types of spatial objects well, and generalizes to higher dimensions. * Bulk loaders exist. Disadvantages: * Can degrade to linear time for operations (why? provide an example). * Performance can degrade if frequent inserts/deletes made (choices of MBRs). Various heuristics can be used to choose which spatial objects to group together in a node. They generally involve tradeoffs in proximity/ empty space coverage/ thinness of MBRS. R-trees are supported in virtually all commercial spatial databases. Lots of approaches have been developed to get around the problem of degenerate R-trees: multiple R-trees designed around the particular application (drones taking hi-resolution images), using the boundary line segments as the objects being stored instead of the entire regions, etc. Variants: R*tree (different splitting and MBR choosing criteria, R+tree)