This paper presents a hierarchical lane detection system with the ability to deal with both structured and unstructured roads. The proposed system classifies the environment first before applying suitable algorithms for different types of roads. For environment classification, pixels with lane-marking colors are extracted as feature points. Eigenvalue Decomposition Regularized Discriminant Analysis is utilized in model selection and maximum likelihood estimation of Gaussian parameters in high dimensional feature space. For structured roads, the extracted feature points are reused for lane detection. For unstructured roads, mean-shift segmentation is applied to divide the scene into regions. Possible road boundary candidates are selected, and Bayes rule is used to choose the most probable boundary pairs. The experimental results show that the system is able to robustly find the boundaries of the lanes on different types of roads and various weather conditions.