Cloud Coverage Prediction Models

1. CLIP Model & its working:

• There are various methods to extract features from images, but here, have utilized the Contrastive Language Image Pretrained (CLIP) Model.
• CLIP is typically used for image-text similarity problems.
• The CLIP model comprises 3 main components: Image Encoder, Text Encoder, and Projection Head.
• Image Encoder: Utilizes ResNet50 to extract features from images in vector format.
• Text Encoder: Utilizes Distilled-Bert to extract features from text in vector format.
• Projection Head: Transforms both image and text vectors into the same size/shape.
• A Dot Product is applied, creating a knowledge base, i.e., image-text mapping knowledge.
• During inference, when an image is passed, image/text vectors are created, and a similarity score is generated between the knowledge base and the image/text vector to provide results.

2. Catboost Model:

• Used Catboost Regressor to calculate cloud coverage.
• Catboost is chosen for its robustness to outliers, high efficiency, out-of-the-box support, and faster processing compared to other bagging and generic models especially when we have image feature vectors.

Technical Workflow Steps:

1. Initially, we have data: images mapped with their cloud coverage values.
2. Generate text from the cloud coverage value, e.g. if cloud coverage = 97, then Text = “There is High Cloud Coverage. Opaque Cloud Coverage is 97%.”
3. Fine-tune the Contrastive Language Image Pretrained (CLIP) model on the skycam images + corresponding text.
4. Extraction of features from skycam images is done using the Fine-tuned CLIP model’s Image Encoder.
5. Extracted features are treated as ‘x’ (independent variables with a size of 2048) and cloud coverage values from Step 1 are treated as ‘y’ (dependent features) for the Catboost model.
6. Training, validation & testing records: 70,168 | 30,072 | 33,414.
7. The best hyperparameters for the Catboost model are determined, and a Regressor is implemented to predict cloud coverage in percentages.
8. Later, developed and deployed a Gradio App on Hugging Face Spaces, where users input a skycam image and receive the opaque cloud coverage percentage as output.

Python Libraries

Lets install following dependencies for the project:

• Keep all the below dependencies in your requirements.txt.
• Run command: pip install -r requirements.txt in your terminal.
• Once dependencies are installed, we are set to go!

numpy
pandas
matplotlib
albumentations
opencv-python
torch
timm
tqdm
scikit-learn
catboost
transformers
gradio
git+https://github.com/openai/CLIP.git

Cloud coverage prediction is critical in weather forecasting and a variety of applications such as solar energy generation, aviation, and climate monitoring. Accurate forecasts help decision-makers and sectors plan for and adapt to changing weather conditions. The advancement of artificial intelligence and computer vision techniques in recent years has created new opportunities for enhancing cloud coverage forecasts.

One promising approach is the use of SkyCam images.

• In the face of rapidly changing global climate patterns, there is an urgent need for innovative tools and technologies to better understand and predict weather-related phenomena.
• One crucial aspect of climate analysis is the assessment of cloud coverage, which plays a pivotal role in influencing weather conditions and climate trends.
• Experts may not always be available to monitor climatic shifts. Therefore, developing an automated weather monitoring system is crucial for various applications, including agriculture and disaster management.

The purpose of this research is to estimate the opaque Cloud Coverage from a Skycam Image using AI/ML methodologies.