Decision Trees

• represent discrete-valued functions

• instances described by attribute-value pairs

• can represent disjunctive expressions (unlike version spaces)

• robust to noisy training data (unlike version spaces)

• can handle missing attributes (unlike version spaces)

• one of the most popular and practical methods of inductive learning

• Idea: ask a series of questions about the attributes of an instance in order to arrive at the correct classification

• classifying patients by disease
• classifying equipment malfunctions by cause
• classifying loan applications by level of risk
• classifying astronomical image data

• learning to fly a Cessna by observing pilots on a simulator
• data generated by watching three skilled human pilots performing a fixed flight plan 30 times each
• new training example created each time pilot took an action by setting a control variable such as thrust or flaps
• 90,000 training examples in all
• each example described by 20 attributes and labelled by the action taken
• decision tree learned by Quinlan's C4.5 system and converted to C code
• C code inserted into flight simulator's control loop so that it could fly the plane by itself
• program learns to fly somewhat better than its teachers
• generalization process cleans up mistakes made by humans
• Reference: Sammut, C., Hurst, S., Kedzier, D., and Michie, D. (1992). Learning to fly. In Proceedings of the Ninth International Conference on Machine Learning, Aberdeen. Morgan Kaufmann.

Example:  Decision tree for the concept PlayTennis

<Outlook=Sunny, Temp=Hot, Humidity=High, Wind=Strong> classified as No
 

In general, decision trees are disjunctions of conjunctions of constraints on attribute values.
 

Above tree is equivalent to the following logical expression:

(Outlook=Sunny ^ Humidity=Normal) v (Outlook=Overcast) v (Outlook=Rain ^ Wind=Weak)

Constructing Decision Trees

Idea: Determine which attribute of the examples makes the most difference in their classification and begin there.
 

Inductive bias: Prefer shallower trees that ask fewer questions.
 

Example: Boolean function: first bit or second bit on.
 

Instance	Classification
A   000	-
B   001	-
C   010	+
D   011	+
E   100	+
F   101	+
G   110	+
H   111	+

We can think of each bit position as an attribute (with value on or off).  Which attribute divides the examples best?

Let's start with Bit1.  The left branch of this tree is complete -- no more work is necessary.  Need to complete right branch.

Consider Bit2 vs. Bit3.  Bit2 is better:

This decision tree is equivalent to the following logical expression:

Bit1(on) v (Bit1(off) ^ Bit2(on))

• does not always find the shortest tree (e.g., Bit1=Bit3 function example)

• inductive bias: shorter trees preferred; trees with high information gain attributes closer to root preferred

Overfitting

• Hypothesis h overfits the data if there exists h' with greater error than h over training examples but less error than h over entire distribution of instances

• Serious problem for all inductive learning methods

• Trees may grow to include irrelevant attributes (e.g., Date)

• Noisy examples may add spurious nodes to tree

• Effects of overfitting

  

Decision Tree Pruning

• Decision trees can be pruned to compensate for overfitting.

• Reduced-error pruning

1. divide data into training set and validation set
2. build a decision tree using the training set, allowing overfitting to occur
3. for each internal node in tree:
• consider effect of removing subtree rooted at node, making it a leaf, and assigning it majority classification for examples at that node
• note performance of pruned tree over validation set
4. remove node that most improves accuracy over validation set
5. repeat steps 3 and 4 until further pruning is harmful

• Effects of reduced-error pruning

  

• Rule post-pruning
• infer decision tree, allowing overfitting to occur
• convert tree to a set of rules (one for each path in tree)
• prune (generalize) each rule by removing any preconditions (attribute tests) that result in improving its accuracy over the validation set
• sort pruned rules by accuracy, and consider them in this order when classifying subsequent instances
• example:

  

  
  IF     (Outlook = Sunny) ^ (Humidity = High)
  THEN   PlayTennis = No
  

  Try removing (Outlook = Sunny) condition or (Humidity = High) condition from the rule and select whichever pruning step leads to the biggest improvement in accuracy on the validation set (or else neither if no improvement results).

• advantages of rule representation:
  • each path's constraints can be pruned independently of other paths in the tree
  • removes distinction between attribute tests that occur near the root and those that occur later in the tree
  • converting to rules improves readability

Continuous-Valued Attributes

Dynamically define new discrete-valued attributes that partition the continuous attribute into a discrete set of intervals

Example:  Suppose we have training examples with the following Temp values:
 

Temp	40	48	60	72	80	90
PlayTennis	No	No	Yes	Yes	Yes	No

Split into two intervals:  Temp < val  and  Temp > val

This defines a new boolean attribute Temp_>val

How to choose val threshold?

Consider boundary cases val = (48+60)/2 = 54 and val = (80+90)/2 = 85

Possible attributes:  Temp_>54 , Temp_>85

Choose attribute with largest information gain (as before):  Temp_>54

This approach can be generalized to multiple thresholds.

Missing Attributes

• If a training example in S is missing an attribute value at some node during tree construction, just "guess" an appropriate value by filling in with the most common value among other examples at that node.

• If a novel instance is missing an attribute value that needs to be tested, determine possible values for that attribute and ask decision tree for classification assuming each possible value.  Then return a final classification based on the individual classifications weighted by the probabilities of each attribute value.

Gain Ratio and Split Information

• Variant of Gain heuristic that penalizes attributes that split the data into many small groups (e.g., Date).

• GainRatio = Gain / SplitInformation

• SplitInformation = Entropy of examples with respect to attribute values.