==================== Trees ==================== Unbalanced Binary Trees =========================== Binary Trees ----------------- .. admonition:: Definition For a binary tree: A tree is a: * empty tree. * an item. * an item followed by :math:`x | 1 \leq x \leq 2` trees. For an n-ary tree: A tree is a: * empty tree. * an item. * an item followed by :math:`x | 1 \leq x \leq n` trees. Note the recursive nature of the definitions. Recursion is a natural fit for trees. A **forest** is a group of 2 or more trees A binary tree API typically has the following operations: * create a tree * isEmpty * contains( value ) * traverse // often through iterators * remove item * add item Tree Traversals: There are 3 classic tree traversals: **pre-order, in-order, and post-order**. They are summarized in the following code snippets: .. code-block:: c preorder(node) { visit(node); preorder(node.leftChild); preorder(node.rightChild); } .. code-block:: c inorder(node) { inorder(node.leftChild); visit(node); inorder(node.rightChild); } If items in the tree are sorted (in a binary search tree), in-order visits items in order .. code-block:: c postorder(node) { postorder(node.leftChild); postorder(node.rightChild); visit(node); } Post-order is useful for printing a tree such that the structure is visible. For the following tree: pre-order:HDBACFEGPKINURY in-order: ABCDEFGHIJNPRUY post-order: ???? .. image:: zstatic/binTree.* :width: 6in .. admonition:: Note about recursion Recursion if often frowned upon in production code. Why? What are the advantages of recursion? What are the disadvantages of recursion? If you need a recursive function, but you don't want some of the disadvantages of recursion (call stack limitations), you can essentially implement the stack yourself. In other words, use an iterative function, but push and pop data onto a stack to simulate recursion. This allows much more efficient stack management (less overhead) and larger stack sizes. A last note. Binary trees can be implemented as arrays rather than node structures.. For example: .. image:: zstatic/binTreeArray.* :width: 3in To compute the children of node with array index :math:`i`: :math:`\mathit{leftChild}(i) = (i*2)+1` :math:`\mathit{rightChild}(i) = (i*2)+2` Array construction of binary trees is super useful for complete or nearly complete trees. One issue to consider is how to tell if a node is empty (essentially, detect a null pointer)? Array construction of binary trees will be super useful later. Binary Search Trees ------------------------- .. admonition:: Definition A binary search tree is a binary tree where each node contains an item such that a total ordering exists over the items. Furthermore, for any node :math:`n` containing item :math:`x`, all items in the left subtree from :math:`n` are less than or equal to :math:`x`, and all items in the right subtree from :math:`n` are greater than :math:`x` Thus, the binary search tree must preserve both tree structure, and the order of the items. Typically, binary search trees are used for searching for items Search operation: Look at a node to see if it has the value we want, if what we are looking for is less than the value, visit the left subtree, otherwise visit the right subtree. Insert operation. Perform a search, but continue until we hit a leaf. Insert the item at the leaf. **inserting at leaves is EASY, inserting in the middle of the tree is hard. we see this a lot with trees** .. admonition:: Exercise Insert the following into a binary search tree. Show the tree at each step: 9, 6, 8, 7, 3, 0, 10, 5 .. image:: zstatic/bst.* :width: 3in Remove operation: The easiest method is the remove from a leaf (just like with insert). So the general idea is copy whatever we want to delete to a leaf, then delete the leaf. Sometimes, we cannot find a leaf, so we must simply find a node with a single child pointer (which is an easy case to handle) One problem, we must exchange the value of a node with a value in a leaf. And, we must do this while preserving the tree ordering (after the delete of course). So, we need to identify either the previous or successive value in the tree with respect to item ordering. Luckily, there is an easy formula to find these. .. admonition:: Finding the next/previous value in sequence in a BST for the delete operation Consider the following tree: .. image:: zstatic/bst.* :width: 3in The value in the tree immediately less than 9 can be found by visiting the left child of the node containing 9, then following right child pointers until a NULL pointer is found The value in the tree immediately greater than 9 can be found by visiting the right child of the node containing 9, then following left child pointers until a NULL pointer is found. Note that in this case, if we want to delete 9, we can swap it with 8, resulting in: .. image:: zstatic/bstDel1.* :width: 3in To delete the node that now contains 9, we can simply connect the right child of the 6 node to the 7 node (because the 9 node only has 1 child): .. image:: zstatic/bstDel2.* :width: 3in The idea is simply to never delete a node with 2 children. Note that the structure and order is maintained in the tree. Finally, the time complexities of BST operations are related to the height of the tree. Why? What are the worst case and best case time complexities of search, insert, and remove? .. admonition:: Balance and BSTs BSTs are NOT guaranteed to be balanced, but, if the input is inserted in a random order, they tend to be. For BSTs, randomness is your friend. Although there is no guarantee of balance, if the input is expected to be random, you can expect logarithmic operation time. However, deleting from a BST tends to degrade randomness an favor more unbalanced trees. BSTs can be a very good and simple choice. There tends to be costs to achieve balance. Balanced Trees ==================== For operations on trees, the worst case cost of the operation is equivalent to the height of the tree. For a BST with items inserted in random order, the costs is roughly :math:`O( \lg n)`. Items inserted roughly in order lead to costs of :math:`O(n)`. Insertions and deletions tend to make things worse (less random). Instead of relying on the randomness of data ordering, lets look at some trees that are guaranteed to be balanced. The guarantee balance comes with a price, these trees are significantly more difficult to implement than traditional BSTs. .. admonition:: Definition: Balanced Tree A balanced tree is a tree that is close to its minimal possible height (its longest branch is close to the minimal possible height for the number of items in the tree). For :math:`n` items, a balanced binary tree will have height close to :math:`\lg n`, a trinary tree will have height close to :math:`log_3 n`, an k-ary tree will have height close to :math:`log_k n`. 2-3 Trees ------------------ A 2-3 tree node has space to hold 2 items and 3 pointers. A node will **always** have 1 item and 2 pointers (a 2 node), or 2 items and 3 pointers (a 3 node): .. image:: zstatic/23Node.* :width: 3in Items in nodes are ordered. So, the first item is always less than or equal to the second item (if there are 2 items in the node). Searching for a value is similar to a BST, except there are 3 possible pointers to follow. The algorithm is as follows .. code-block:: python find( node*, item): is the item in the first slot? if Yes: done if No: Is the item less than item in first slot? if Yes: find( node.leftChild ) if No: Is the item in second slot? if Yes: done if No: is the item less than item in second slot? if Yes: find( middleChild, item ) if No: find( rightChild, item ) **Inserting into a 2-3 tree** The 2-3 tree is guaranteed to ALWAYS be balanced. Therefore, we cannot simply insert items by creating new leaves like we do with a BST. The process is as follows: * Always insert a new item at a leaf, but never increase the depth of the tree at the leaves. * Use the search algorithm to find the leaf that a new item should go in. * If it is a 2 node, then just put the item there. * If is a 3 node, split the node into 2 nodes, and promote the middle of the 3 values up to the parent node. * If the parent does not have room (it is already a 3 node), split that node and promote the middle value to its parent. Recursivley repeat until no splittting occurs. If this procedure is followed, the height of the tree will be increased at the TOP of the tree (the root). Basically, tree height will increase once a root is split. An example follows: .. admonition:: Example 2-3 Tree Insertion Insert the following: 10 40 20 5 15 17 19 .. image:: zstatic/23Insert.* :width: 6in On paper, the insertion algorithm is not too bad. By drawing pictures, you can usually figure out where things should go. Actually implementing the code is anoother story. The following is a working 2-3 tree implemenation. Try to summarize the portions of the code in pseudo-code. Make sure you know what each code block is doing. The downloadable source file is :download:`here ` .. literalinclude:: zstatic/t23.py :language: python :linenos: **Delete:** Deleteing from a 2-3 tree is super tricky. Consider the following example. We can't delete in that fashion because it breaks the search algorithm! .. image:: zstatic/t23delete.* :width: 6in The basic idea to delete from a 2-3 tree is to do something similar to BST delete, but to enforce balancing. Basically, move the item to be deleted to a leaf node. Thenremove the item. Then, if a node is empty, recursively merge nodes up the tree (possible shortening the tree). Merging nodes is a pain, and there a lot of special cases to code. Typically, you try to avoid merging by stealing data from other nodes, but this sut leads to more special cases to code. We will not focus on 23 tree delete. **Roundup:** * Pro: 2-3 trees are balanced! insert, search, delete are all :math:`O(\lg n)`. * Con: Super hard to implement, especially delete. * Con: Nodes are not all full (wasted space?), but we get a balanced tree. * Con: Hard to debug. So, 2-3 trees are a nice introduction to balanced trees because they succinctly summarize the tradeoffs faced by all balanced trees: balanced trees are HARD to implement (especially fully with delete, etc). But, we get good guaranteed worst case performance! 2-3-4 Trees ----------------- One reason that 2-3 trees are so difficult is that is that when splitting/combining nodes, we end up having 3 items to deal with, but only space for 2 in a single node. This just makes things more difficult. 2-3-4 trees are similar to 2-3 trees, but allow 1, 2, or 3 items per node, alleviating that particular problem. Furthermore, we will implement the splitting strategy with a different philosophy to avoid recursive node splitting, greatly simplifying the code. .. admonition:: 2-3-4 Tree Definition: A 2-3-4 tree is a height balanced search tree such that the leaves of the tree are all at the same depth. A node can contain 1, 2, or 3 items and 2, 3, or 4 pointers. Items in a node are sorted, and an item is always surrounded by valid pointers (except for leaf nodes). The node is structured as follows: .. code-block:: python class node: def __init__(self): self.data = [None, None, None] self.ptr = [None, None, None, None] The height of the tree will be somewhere between :math:`O(\lg n)` and :math:`O(\log_4 n)`: * Balanced binary trees :math:`\rightarrow` height :math:`\lg n` * Balanced tri-nary trees :math:`\rightarrow` height :math:`\log_3 n` * Balanced quad-nary trees :math:`\rightarrow` height :math:`\log_4 n` .. admonition:: Insert Algorithm: Like a 2-3 tree, always insert at a leaf. Recall that inserting into a 2-3 tree is complicated mainly because of the recursive code. We will take a different philosophy for 2-3-4 trees. Instead of recursively splitting, we will preemptively split full nodes as we descend the tree, so we will **NEVER** have to recursively split a node. Thus, the algorithm is as follows: .. code-block:: python Get the root node If the root is full: split the root, assign items and pointers Get the pointer P to the appropriate child node. while P is not null: if P is full: split the node pointed to by P, assign items and pointers if P is a leaf: add the item to the appropriate spot in P (it will always have room) P is assigned NULL (break out of the loop) else: Get the pointer P to the appropriate child node Lets see an example: .. image:: zstatic/t234insert1.* :width: 6in .. image:: zstatic/t234insert2.* :width: 6in .. image:: zstatic/t234insert3.* :width: 6in .. admonition:: Exercise: Implement 2-3-4 Tree Insert Here is a skeleton to get you started. I have essentially implemented the easy parts of the pseudocode above, and left the hard parts to you. One thing to note is that because we preemptively split down the tree, there are only 2 cases that need to be handled for insert. .. code-block:: python :linenos: class node: def __init__(self): self.data = [None, None, None] self.ptr = [None, None, None, None] def isLeaf(self): return self.ptr[0] == None def getDataIndex( self, data ): if self.data[0] == None: return 0 elif data <= self.data[0]: return 0 elif self.data[1] == None: return 1 elif data <= self.data[1]: return 1 else: return 2 def getPtrIndex( self, data ): if data <= self.data[0]: return 0 elif self.data[1] == None: return 1 elif data <= self.data[1]: return 1 elif self.data[2] == None: return 2 elif data <= self.data[2]: return 2 else: return 3 class t234: def __init__( self ): self.root = node() def splitRoot(self, nodePtr): if nodePtr.data[2] == None: return nodePtr # Your code here return newRoot def split( self, nodePtr, index ): if nodePtr.ptr[index].data[2] == None: return # Your code here def insert( self, data ): #check the split for root self.root = self.splitRoot( self.root ) curr = self.root while curr != None: #check if we are at a leaf if curr.isLeaf(): index = curr.getDataIndex( data ) #for i in range( len( curr.data)-1, index, -1 ): # curr.data[i] = curr.data[i-1] curr.data[(index+1):] = curr.data[index:2] curr.data[index] = data return else: #find pointer to follow index = curr.getPtrIndex( data ) self.split( curr, index ) index = curr.getPtrIndex( data ) curr = curr.ptr[index] def printTree( self ): self.printRecursive( self.root, 0 ) def printRecursive( self, curr, depth ): #print right to left, then curr if curr == None: return # Finish the print function testWords = ['banana', 'apple', 'carrot', 'lettuce', 'pepper', 'kljdfg', 'sdflkj', 'fdgijerl', 'reoijht', 'troiu', 'ertoij', 'erwt', 'erty', 'ezz', 'exa', 'oerijfgdkm', 'sdff', 'qwer'] print testWords print sorted( testWords ) **DELETE** Deleting from a 234 tree is, again, hard. The idea, again, is to move the item to delete to a leaf, then remove it from the leaf. So, you swap the item to delete with its in-order successor. Then delete the item. If the node is then empty, you must merge nodes, again maintaining the tree properties of being perfectly balanced. Because you have 3 items in a node, it is possible to steal items from a neighbor. **Roundup:** The 234 tree is easier to implement than a 2-3 tree, but still tough, especially with delete. Again, there are lots of special cases with the delete algorithm that must be handled. * Pro: height balanced, so good performance * Con: increased memory usage (possibly 2 empty items and 3 empty pointers in every node) * Con: still hard to debug B+ trees ------------------------ **Disk Architecture Review** The hard disk is considered part of the memory hierarchy in computers (it is used to implement virtual memory). A typical memory hierarchy is: .. image:: zstatic/tmemHierarchy.* :width: 5in Hard disks are typically the rotating magnetic platter type (SSDs are too expensive to completely replace them). Thus, HDs are the only mechanical device in your computer (other than fans and the DVD drive), and thus are the SLOWEST! HD access times are so long that computer users notice it takes a long time (think booting up the computer). Because HD access times are so long, any data structure that needs to work well in a disk environment must be designed specifically for disk interaction. Purpose: * Long term, nonvolatile storage * Lowest level in the memory hierarchy * slow, large, inexpensive General structure * A rotating platter coated with a magnetic surface * A moveable read/write head to access the information on the disk Typical numbers: * 1 to 4 platters (each with 2 recordable surfaces) per disk of 1” to 3.5” in diameter * Rotational speeds of 5,400 to 15,000 RPM * 10,000 to 50,000 tracks per surface * cylinder - all the tracks under the head at a given point on all surfaces * 100 to 500 sectors per track. The smallest unit that can be read/written. .. image:: zstatic/tdisk1.* :width: 4in .. image:: zstatic/tdisk2.* :width: 4in **B+ trees:** B trees and (more commonly used) B+ trees are a generalization of 2-3-4 trees to nodes with larger amounts of items and pointers. They are typically used for external memory search trees (in DATABASES!). Thus, they are typically referred to as an **index structure**. An index structure is simply database terminology for a data structure that provides faster data access than simply iterating through all items in a collection. B+ trees provide: 1. An index structure that is seperate from the data. 2. Hard lower and upper bounds on size and time to complete operations 3. Design specific to disk based structures. B trees and their variants are used in virtually every DBMS out there. We need a concept of a **key** that uniquely defines a record (basically an item). We will store complete records in one file (the relation). We will create a new file to hold the index. The index will contain keys that identify records, and the **disk address** of that record in the relation's file. This means we don't need to re-order the relation's data file ever, we can simply append to it. Also, we can create as many index files as we want for whatever we want to search on. We will talk about B+ trees, they are easier to use and manage, and work more naturally. .. admonition:: B+Tree A B+tree contains 2 types of nodes: **internal** nodes and **leaf** nodes. - internal nodes contain keys and pointers to other internal nodes or leaf nodes. - leaf nodes contain keys and disk addresses of records in the relation's data file. A node is **fixed size**. Usually, a node is the size of a disk page. Sometimes it will be a multiple of the size of a disk page. This means the number of items stored in a node is dependent on the size of the item, and the size of the disk page (usually 4k). let :math:`b` be the *branching factor* of nodes in a B+tree. :math:`b` indicates the number of child nodes for a node in the tree. Let :math:`m` be the actual number of keys in a node. For internal nodes :math:`\lceil b/2 \rceil \leq m \leq b`. For leaf nodes: :math:`\lfloor b/2 \rfloor \leq m \leq b-1`. Exceptions are made for the root node if there are simply aren't enough items in the tree to reach the minimum threshhold. **Node Structure**: Each node contains the following structure: [pointer, key, pointer, key, pointer, ..., key]. Pointers in internal nodes point to other nodes. Pointers in leaf nodes contain disk addresses, expect for the last pointer which points to the next successive node to the "right" in the tree **Note** in B+ trees, keys are *repeated* in internal nodes and in leaves. This means **every** key will show up in a leaf. Some keys will show up in internal nodes. This is a major difference with standard B Trees .. image:: zstatic/tBplus.* :width: 6in **Searching the tree**. Search the tree like a binary search tree **Insertion Into the Tree** 1. Find the node at which to insert the item 2. If there is room in the node, insert it. 3. If there is no room, find the median value :math:`x`. 4. Create two new nodes, place all keys :math:`k\leq x` in the left node, and all keys :math:`k>x` in the right node. 5. Move the median value :math:`x` up to the parent node of the split node, and connect the tree. If there is no room for the median value in the parent node, repeat steps 4-5 on that node B and B+trees are *complete trees*. They do not grow at the leaves, instead, they grow from the *root*! .. admonition:: Example Put the following keys into a B Tree and a B+Tree. Let the branching factor :math:`b=4`: 20 5 8 40 23 31 1 10 33 41 22 7 16 44 27 35 11 9 2 17 How many nodes are visited to find the value 20? 33? 44? How many nodes are visted to traverse the tree starting with value 31? what is the time complexity of search and insert? The algorithm for searching and inserting into a B+tree is straightforward, but the implementation of insertion can be tricky. In the worst case, a split can propogate up the entire tree to the root. It is useful to keep track of a stack of node pointers that lead back from the node of insertion ot the root. As with most trees, the time complexities of the tree are dependent upon th tree height, for B+trees, the tree is **always** balanced, but trees with more full nodes might be shorter than trees holding the same amount of data with less full nodes. For example, a B+tree with branching factor :math:`b` and height :math:`h` can have : - at most :math:`b^h` pointers to data records. why? - at min :math:`\lceil b/2 \rceil^h` pointers to data records. why? - a height bound for :math:`n` elements of :math:`O(\lg_b n)`, meaning logarithmic search, insert and delete! .. admonition:: Example Assume a 4096 (4Kb) disk page and records with integer keys (4 bytes). Assume a 64 bit system (8 byte memory pointers). If B+tree blocks contain no header information, the we have :math:`(4096-8)/(4+8) = 340` keys per block (1 pointer and 1 key takes 12 bytes. the -8 is for the last pointer). Lets assume a typical block will have 255 keys (somewhere between 340 and 340/2=170). - With a root, we can store 255 pointers to records - With a root and 255 leaves, we have pointers to :math:`255^2 = 65,025` records - with a root and 255 internal nodes and 255 leaves for each internal node, we have pointers to :math:`255^3 = 16.6 mil` records So, such a B+tree with height 3 requires 3 disk accesses to search 16.6 mil records! Furthermore, the top few levels of a B+tre are typically buffered in memory, eliminating the need to disk access in may cases. **B+ Tree Delete** Again, like the 23 and 234 trees, delete is hard. In fact, because B+trees are used in databases, delete is often not worth it to implement. Instead, we use a **lazy delete** scheme. Instead of deleting and re-balancing, just mark an item as being deleted, and hopefully reuse the space later. Databases are in the business of remembering information, so delete should not happen too often. Think of an example. A problem with this is wasted space, but hard drives are CHEAP!, so it is not such a problem. **I/O Complexity** We now have a new type of complexity, I/O complexity. Because disk access takes so long, the point is to have few I/O operations, and not worry too much about the in-memory stuff. So, for a B+ tree with :math:`a` min nodes and :math:`b` max nodes and block size (disk page size) of :math:`B`: * Number of leaf blocks is :math:`O(n/B)` * IO complexity for all operations is :math:`O(\log_B n)` * Height of tree is :math:`\Omega (\log_a n) \mathit{~and~} O(\log_b n)` * Time complexity to find is between :math:`\Omega( f(a) \log_a n ) \mathit{~and~} O( f(b) \log_b n )` * Where :math:`f(b)` is the time to find an element in a node **A final note when dealing with data that is big: Always remember your bandwidth** http://hothardware.com/News/Homing-Pigeon-Faster-Than-Internet-in-Data-Transfer/ .. code-block:: bash Time to transfer 4GB at 2.04MB per second is …… 4 hours, 39 minutes, and 37 sec Time to transfer 2.57 PB == 2570000GB at 2.04Mbits per second is 130821 Days 12 Hours 32 Minutes 13.54 Seconds == 358 years! Size of a hard drive: .01 cubic foot Cargo capacity of a Toyota Yaris: 25.7 cubic feet Number of hard drives I can transport: 2570 If these are 1 TB hard drives, that’s 2.57 PB == roughly 20.56 peta bits Time to drive to Chicago: 5hrs == 18000 seconds Which gives a bandwidth of 1.14 Tbits/second == 142 GB/second And so the saying is: “Never underestimate the bandwidth of a station wagon loaded with hard drives hurtling down the highway at 70mph” Red Black Trees (RBTs) --------------------------- 2-3-4 trees are nice, but have some significant drawbacks: 1. they are hard to implement fully. 2. they have potentially a lot of wasted space in the nodes 3. delete is still a nightmare A nice idea is to have the 2-3-4 tree advantages in a binary tree format. This is the idea behind **red-black trees**. Basically, for every 234 tree, there is an equivalent RBT. Basically, we are going to split up the 3 and 4 nodes into a group of binary nodes. The connected binary nodes are said to be connected with **RED** links, and the other links are **black links**. For example, a mapping between 234 and RB nodes follows: .. image:: zstatic/trbNodes.* :width: 6in **Note**: because red links are ,in some sense, internal node links in the 2-3-4 tree, there can never be 2 consecutive red links on a root to leaf path through the tree! **Note**: black links are pointers in the 2-3-4 tree, so if we compress the red links to re-create the 2-3-4 tree, all the tree height properties still remain: .. admonition:: Properties .. image:: zstatic/trbStruc.* :width: 5in :math:`n:=` number of nodes :math:`H:=` tree height :math:`B:=` height of tree with red links compressed (height of the equivalent 2-3-4 tree) * Property: :math:`2^B \leq n+1 \leq 4^B` * Property: :math:`\frac{1}{2}\lg(n+1) \leq B \leq \log(n+1)` * Property: :math:`\lg(n+1) \leq H \leq 2\lg(n+1)` **So, tree height is** :math:`O(\lg n)` With this style of implementation, the drawbacks to 2-3-4 trees (mostly) disappear. At least the wasted space disappears. Implementation is non-trivial, but easier than 2-3-4 trees. So, a RBT node must contain an extra bit of information that records the color of the incoming link to each node. **Insertion Algorithm** .. code-block:: bash Perform an insertion identical to binary search tree insertion (insert at the leaf! just like a 2-3-4 tree) Color the incoming link to the newly inserted node RED If the parent node of the newly inserted node also has a RED link coming into it, then we have 2 consecutive red edges Fix the problem using rotations and promotion! .. admonition:: Fixing the Tree! If the tree is unbalanced after an insertion, we perform rotations and promotions to fix it. Here are the cases: Key: * G is the grandparent of the inserted node * P is the parent of the inserted node * N is the newly inserted node Case 1: No violation .. image:: zstatic/trbRot1.* :width: 3in Case 2: The incoming edge of **P** is red and its sibling is black .. image:: zstatic/trbRot2.* :width: 6in Case 2.5: Same as Case 2, but the value of **N** falls in between the value of **P** and the value of **G** in sequence .. image:: zstatic/trbRot3.* :width: 6in Case 3: The incoming edge of **P** is red and its sibling is also red. This is called a promotion. This will only occur after a rotation has already taken place. It may have to be recursively repeated up the tree. .. image:: zstatic/trbRot4.* :width: 6in Summary: If 2 consecutive red edges are present immediately after an insertion, perform a rotation to restructure the tree. Only ONE rotation will be necessary. After that, traverse the path to the root and perform promotions if you come across any consecutive red edges. * only 1 rotation per insertion * promotions may continue up the tree (:math:`O(\lg n)` times) * time complexity is :math:`O(\lg n)` .. admonition:: Example of Insertion Insert the following numbers in this order: 10, 17, 15, 12, 20, 19, 18 .. image:: zstatic/trbExample.* :width: 6in As an exercise, identify the nodes in the equivalent 2-3-4 tree! .. admonition:: Delete from a RBT Use the delete algorithm for a simple BST If the node deleted was a red node, then we are done (we removed an item from a 3 or 4 node in the equivalent 234 tree) If the node deleted was a black node (the incoming link is black), then we deleted a node from the equivalent 234 tree, and the entire thing is now unbalanced. We must rebalance the tree. Re-balancing after a delete is similar to re-balancing after an insert, we do at most 1 restructuring, then we do recolorings (demotions instead of promotions) up the tree. AVL Trees ---------------- AVL trees are very similar to red black trees, except that each node contains a number indicating its balancing factor rather than a boolean indicating a color value. If a node becomes unbalanced, then rotations, identical to RBT rotations, are used to recursively re-balance the tree. Like the other balanced trees, operations are :math:`O(\lg n)` A very good C implentation of an AVL tree exists in the wild as the gnuAVL library. If you need a height balance binary search tree, and can't find a built in language solution, I recommend it. Splay Trees ----------------- Splay trees are an interesting tree. They are not guranteed to be balanced, but they tend to be. Recall that that BST created with insertions of numbers in a random order tends to be balanced. Splay trees try to introduce randomness into a BST by performing rotations (similar to RBT rotations) when items are accessed. They also have a nice philisophical approach to finding items even faster than a typical BST. .. admonition:: Thought... What does computer **cache** do? Why? Why does it work? Does this hold for data structures? The basic premise of the splay tree is that recently accessed items should percolate to the top of the tree (for the same reasons as cache). In order to do this, every time an item is accessed (inserted, searched for, etc.), rotations occur on the tree to raise the item to the root. The rotations cause the tree to shuffle its structure, leading to a tree that is usually pretty close to being balanced, but not guaranteed to be. The rotations used are as follows: .. admonition:: Splay Tree Rotations A single rotation (similar to red black tree rotation), known as a **ZIG**: .. image:: zStatic/tSplayRot1.* :width: 3in A **ZIG-ZIG** double rotation: .. image:: zStatic/tSplayRot2.* :width: 3in A **ZIG-ZAG** double rotation: .. image:: zStatic/tSplayRot3.* :width: 3in Note that there are symmetric left/right versions of each rotation. A singe ZIG only occurs if no grandparent node exists (the parent is the root of the tree). Here are two examples: .. image:: zStatic/tSplayEx0.* :width: 6in .. image:: zStatic/tSplayEx1.* :width: 6in Because splay trees tend to balanced, their operations tend to be :math:`O( \lg n)`. However, proving that a splay tree always performs within a constant factor of other height balanced binary search trees is an open problem. * Pro: Easy to implement, no bookkeeping, just always bubble items to the top * Pro: Tends to behave well (fast access times) * Con: cannot guarantee :math:`O( \lg n)`, but we usually get it. * Con: cannot gurantee balance. Last word on Delete: Lazy Delete! -------------------------------------- As we have seen, delete is often hard to implement and frequently it is just not worth it. Delete causes the following problems: 1. Its hard to implement and debug 2. Tree restructuring can be expensive (especially for B+ trees on disk, but also in terms of memory operations on in memory trees). Also, changing the tree structure means other threads cannot access the tree while the restructuring is occurring. Instead, we can relax the rules a little, and just mark an item as being deleted instead of actually deleting it. This allows us to reuse the space on future inserts. Be careful, we cannot simply destroy the value, because it might be needed to properly search the tree. This is why we simply mark it as deleted. Try an example on one of the trees we have examined so far (234 trees are good for this). * Pro: easier to implement, but must maintain deleted values until they are overwritten * Con: possibly wasted space. Tries ======================== So far we have been focusing very intently on balanced trees. However, general purpose trees are not always the answer. We can always do better at an application if we can make assumptions about what is and isn't allowed. The only requirement for search trees is that the items must have a total ordering and less than and an equal operation defined for them. But, what if what we want to store is text, or any data in a list form. Tries are designed to store items construscted as a list of symbols from an alphabet. The most intuitive example is that tries are superuseful for storing strings. .. admonition:: A couple of asides: 1. The name trie comes from its us for re-**trie**-val, and so is pronounced like *tree* by some and like *try* by others. I will pronounce them like *try* to avoid confusion with other trees. 2. Implementations vary 3. Tries store any data items such that items are constructed as a list of symbols from some alphabet. Tries are very fast for lookup and can exploit redundancy in items to save space. We will look at the following: 1. Standard tries 2. Compressed tries (eliminating chains of nodes) 3. Compact tries (storing array indices instead of symbols) 4. Suffix tries (storing all suffixes of a string) **Standard Tries** The basic idea is that a node only contains a boolean value, and the pointer represents the character being stored. So, a string is constructed by examining its sequence of pointers throughout the data structure. For example, the following image shows a trie containing the words a, an, and, any, at. A node will have a boolean value of TRUE if a word ends at that node, and FALSE otherwise: .. image:: zstatic/tTrie1.* :width: 4in To find out if the word "an" is stored in the trie, we go the root and see if the "a" pointer is not NULL. We then follow the "a" pointer to another node and see if the "n" pointer is not NULL. We then follow the "n" pointer and see if the node has a value of TRUE, indicating that a word ends at that node. Thus, for a trie contianing lowercase character strings, a node is structured as follows: .. code-block:: python class node: def __init__( self ): self.wordEndsHere = False self.letterPointers = [None] * 26 # 26 pointers Let: * n = total size of the strings to be indexed by the trie * d = length of the string being inserted/ searched for/ deleted in an operation * m = the size of the alphabet (remember case, puntuation, whitespace, numbers, etc) Standard tries use: * :math:`O(n)` space * :math:`O(d \times m )` time for search/insert/delete So they are super efficient! An example, a word matching trie: .. image:: zstatic/tTrieWordMatch.* :width: 6in .. admonition:: Watch out for memory! For a true with only lowercase letters, it has 26 pointers per node. A complete trie of depth 6 will have: * :math:`26^1` pointers at level 1 * :math:`26^2` pointers at level 2 * :math:`26^3` pointers at level 3 * :math:`26^4` pointers at level 4 * :math:`26^5` pointers at level 5 * :math:`26^6` pointers at level 6 So, at level 6, we have :math:`26^6=308,915,776 *\frac{8 ~bytes}{1 ~pointer} = 2,471,326,208 = 2GB` of pointers at level 6 alone! this can be problematic **Compressed Tries** Observation, if we have a chain of nodes with only 1 pointer, we can simply compress them into a single node: .. image:: zstatic/tTrieCompressed.* :width: 6in **Compact Tries** A compact representation of a compressed trie. Instead of storing all the symbols in a node, just store array indices: .. image:: zstatic/tTrieCompact.* :width: 6in **Suffix Trie** A compressed trie of all suffixes of a piece of text. Its useful for finding patterns in any part of of text Example: IPDPS. Suffixes: * IPDPS * DPS * PDPS * PS * S .. image:: zstatic/tTrieSuffix.* :width: 3in Let: * n = length of a string X * m = Alphabet size * d = length of Pattern P Properties: * Uses O(n) space * Can be constructed in O(n) time * Find pattern P in X in :math:`O(d \times m)` time (Proportional to length of pattern, not text) A suffix trie example: .. image:: zstatic/tTrieSuff2.* :width: 6in **Some uses of tries** * Search engine index: A collection of all searchable words stored in compressed trie. Each leaf of trie is associated with a word and a list of pages (URLs) containing that word (called occurrence list). Trie is kept in memory (fast). Occurrence lists kept in external memory and ranked by relevance. * Computational Biology: DNA: sequences of 4 different nucleotides (ATCG). Portions of the sequence produce proteins, etc. Its useful to find patterns. Genome: Master DNA sequence for an organism. For humans: 46 chromosomes and 3 billion nucleotides. Roundup ===================== Trees are great and easy to implement, but can degenerate into linear structures. Height balanced trees are great and guarantee logarithmic time operations (for operations based on the height of the tree), but are HARD to implement. If we make some assumptions about our data/application area, we can get away with not exactly height balanced trees (like splay trees that are easy to implement), or something like tries that are easy to implement and incredibly fast in terms of operation speed. For each of the following data structures, you should be able to: 1. list the best case and worst case time complexities of the insert, search, and delete operation. 2. list the best case and worst case time complexity of iterating over all elements in the structure. 3. list pros and cons for each structure. 4. Given a scenario, you should be able to justify which structure you would use (or the inverse, provide scenarios that a particular structure is well suited to, including any assumptions you must make about the scenario). 5. Perform insertions and searches on the data structure. 6. Understand the approach to performing deletes on the structures, and be able to do it on paper (not be able to reproduce the algorithm) * Unbalanced Binary Trees 1. Binary Trees 2. Binary Search Trees * Balanced Trees 1. 2-3 Trees 2. 2-3-4 Trees 3. B+ trees 4. Red Black Trees (RBTs) 5. AVL Trees 6. Splay Trees * Tries