This paper presents a systematic approach to handling building energy consumption data for the prediction of next-day daily energy consumption and peak power demand. Firstly, outlier detection, which relies on the feature extraction of daily energy consumption profiles and the generalized extreme studentized deviate (GESD), is performed to remove the abnormal daily energy consumption profiles. Secondly, the recursive feature elimination (RFE) is adopted to select the optimal input variables for the development of support vector regression (SVR) models. Thirdly, to ensure the generalization performance, leave-group-out cross validation (LGOCV) is conducted to determine model parameters. The proposed outlier detection method is proved to have solid performance in identification of abnormal daily energy consumption profiles. The RFE process is shown to be effective in variable selection and enhancing the model performance. The prediction performance is evaluated using testing data. The accuracies are 3.04% and 3.72% for the next-day daily energy consumption and peak power demand, respectively. The proposed approach can be used to facilitate the decision-making in the building energy field, such as fault detection, selection of operating strategy, and dynamic pricing in smart-grid.