This study introduces a vision-based deep learning approach aimed at enhancing the control of Heating, Ventilation, and Air-Conditioning (HVAC) systems by detecting and recognizing occupant activities and window opening behaviour in real-time. Traditional HVAC control methods often fall short in responding to the dynamic behaviours of occupants, leading to energy inefficiencies and compromised indoor air quality. The motivation behind this research arises from the pressing need for energy-efficient buildings and sustainable practices. Leveraging deep learning and computer vision techniques, the proposed methodology involves the development and training of a specialized convolutional neural network (CNN) model using diverse video data capturing occupant actions and window operations. Rigorous experimental evaluations conducted within a case study building demonstrate impressive detection accuracies of 92.72% for occupant activities and 87.74% for window operations, validating the model's ability to accurately identify a wide range of occupant behaviours, including subtle window-opening actions. By utilizing real-time detection, Deep Learning Influenced Profiles (DLIP) are generated, allowing for building energy simulations to assess the approach's impact on energy demand and indoor air quality. The implications of this vision-based deep learning approach are far-reaching, as it holds promise in predicting indoor air quality, enhancing thermal comfort, and optimizing HVAC and natural ventilation systems based on real-time occupant behaviour, thereby contributing to energy conservation and fostering a more sustainable and comfortable indoor environment.