Part II. UI Inference Codes for Deployed Model

A. cloud_coverage_pipeline.py code:

The provided below code comprises several essential components for cloud coverage prediction.
It begins by importing necessary libraries, including popular deep learning frameworks like PyTorch and Hugging Face Transformers.
The configuration settings (CFG) are specified for hyperparameters, dataset, and model configurations.
The CLIP Model class is defined to encapsulate the Cloud Coverage Prediction model, incorporating the Image Encoder and optional Text Encoder and Projection Head modules.
The Image Encoder class employs a pre-trained ResNet model from the ‘timm’ library to extract image features.
The Sky Image class defines a custom dataset loader for image data, allowing for data transformation and preprocessing. It takes images as input.
Additional utility functions are included for initializing models (Catboost and CLIP), extracting features from the CLIP model, and predicting cloud coverage based on the extracted features.
Overall, the code sets up the foundation for cloud coverage prediction, including data loading, model initialization, and feature extraction, making it ready for cloud coverage assessment using the CLIP and Catboost models.
This file returns predicted cloud coverage in percentage to app.py which returns the cloud coverage to the User on User Interface.

Python3

# Importing Libraries
import os
import numpy as np
import cv2
import torch
from torch import nn
import timm
import pickle
from transformers import DistilBertModel, DistilBertConfig
from torch.utils.data import Dataset, DataLoader
from tqdm.autonotebook import tqdm
os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
 
 
# Trained Model Configurations
CFG = {
    "debug": False,
    "captions_path": ".",
    "batch_size": 64,
    "num_workers": 4,
    "head_lr": 1e-3,
    "image_encoder_lr": 1e-4,
    "text_encoder_lr": 1e-5,
    "weight_decay": 1e-3,
    "patience": 1,
    "factor": 0.8,
    "epochs": 12,
    "device": "cpu",
    "model_name": 'resnet50',
    "image_embedding": 2048,
    "text_encoder_model": "distilbert-base-uncased",
    "text_embedding": 768,
    "text_tokenizer": "distilbert-base-uncased",
    "max_length": 200,
    "pretrained": True,
    "trainable": True,
    "temperature": 1.0,
    "size": 224,
    "num_projection_layers": 1,
    "projection_dim": 256,
    "dropout": 0.1
}
 
 
# Loading Finetuned Clip Model to the below class format
class CLIPModel(nn.Module):
    def __init__(
            self,
            temperature=CFG["temperature"],
            image_embedding=CFG["image_embedding"],
            text_embedding=CFG["text_embedding"],
    ):
        super().__init__()
        self.image_encoder = ImageEncoder()
        self.text_encoder = TextEncoder()
        self.image_projection = ProjectionHead(embedding_dim=image_embedding)
        self.text_projection = ProjectionHead(embedding_dim=text_embedding)
        self.temperature = temperature
 
 
# Image Encoder Class to extract features using finetuned clip's Resnet Image Encoder
class ImageEncoder(nn.Module):
    def __init__(self, model_name=CFG["model_name"], pretrained=CFG["pretrained"], trainable=CFG["trainable"]):
        super().__init__()
        self.model = timm.create_model(
            model_name, pretrained, num_classes=0, global_pool="avg")
        for p in self.model.parameters():
            p.requires_grad = trainable
 
    def forward(self, x):
        return self.model(x)
 
 
# Text Encoder - Optional in inference
class TextEncoder(nn.Module):
    def __init__(self, model_name=CFG["text_encoder_model"], pretrained=CFG["pretrained"], trainable=CFG["trainable"]):
        super().__init__()
        if pretrained:
            self.model = DistilBertModel.from_pretrained(model_name)
        else:
            self.model = DistilBertModel(config=DistilBertConfig())
 
        for p in self.model.parameters():
            p.requires_grad = trainable
 
        self.target_token_idx = 0
 
    def forward(self, input_ids, attention_mask):
        output = self.model(input_ids=input_ids, attention_mask=attention_mask)
        last_hidden_state = output.last_hidden_state
        return last_hidden_state[:, self.target_token_idx, :]
 
 
# Projection Class - Optional in inference
class ProjectionHead(nn.Module):
    def __init__(
            self,
            embedding_dim,
            projection_dim=CFG["projection_dim"],
            dropout=CFG["dropout"]
    ):
        super().__init__()
        self.projection = nn.Linear(embedding_dim, projection_dim)
        self.gelu = nn.GELU()
        self.fc = nn.Linear(projection_dim, projection_dim)
        self.dropout = nn.Dropout(dropout)
        self.layer_norm = nn.LayerNorm(projection_dim)
 
    def forward(self, x):
        projected = self.projection(x)
        x = self.gelu(projected)
        x = self.fc(x)
        x = self.dropout(x)
        x = x + projected
        x = self.layer_norm(x)
        return x
 
 
# Class to transform image to custom data format
class SkyImage(Dataset):
    def __init__(self, img, label):
        self.img = img
        self.img_label = label
 
    def __len__(self):
        return len(self.img)
 
    def __getitem__(self, idx):
        image = cv2.resize(self.img[idx], (244, 244))
        image = np.moveaxis(image, -1, 0)
        label = self.img_label[idx]
        return image, label
 
 
# Method to initialize CatBoost model
def initialize_models():
    cbt_model = pickle.load(open("/home/gfg19509@gfg.w3wiki.org/PawanKrGunjan/Computer Vision/Skycam/catboost_model.sav", 'rb'))
    clip_model = CLIPModel().to(CFG["device"])
    clip_model.load_state_dict(torch.load(
        "/home/gfg19509@gfg.w3wiki.org/PawanKrGunjan/Computer Vision/Skycam/clip_model.pt", map_location=CFG["device"]))
    clip_model.eval()
    return cbt_model, clip_model
 
 
# Method to extract features from finetuned clip model
def get_features(clip_model, dataset):
    features, label, embeddings = [], [], []
    with torch.no_grad():
        for images, labels in tqdm(DataLoader(dataset, batch_size=64)):
            image_input = torch.tensor(np.stack(images)).cpu().float()
            image_features = clip_model.image_encoder(image_input)
            features.append(image_features)
            label.append(labels)
    return torch.cat(features), torch.cat(label).cpu()
 
 
# Method to calculate cloud coverage
def predict_cloud_coverage(image, clip_model, CTBR_model):
    img, lbl = [image], [0]
    # Transforming Data into custom format
    test_image = SkyImage(img, lbl)
    # Extracting Features from Finetuned CLIP model
    features, label = get_features(clip_model, test_image)
    # Predicting Cloud Coverage based on extracted features
    pred_cloud_coverage = CTBR_model.predict(features.cpu().numpy())
    return round(max(0.0, min(100.0, pred_cloud_coverage[0])), 1)

app.py code:

The below code sets up a Gradio web interface for a cloud coverage prediction model.

It imports necessary libraries, initializes the CLIP and Catboost models, and defines a predict function to calculate cloud coverage based on an uploaded sky image.
The Gradio app takes an image as input, processes it, and provides a textual prediction output, categorizing cloud coverage as low, moderate, or high.
The interface allows users to upload sky images and receive cloud coverage predictions interactively.
The app launches and runs for real-time predictions via a web interface.
There is a separate cloud_coverage_pipeline.py file which includes pipeline code. Keep it in same folder.

Python3

# Importing Libraries
import gradio as gr
from gradio.components import Image, Textbox
from cloud_coverage_pipeline import predict_cloud_coverage, initialize_models
 
# Initialize the CLIP model and CatBoost model only once
cbt_model, clip_model = initialize_models()
 
# Method to call cloud_coverage_pipeline.py to calculate cloud coverage
 
 
def predict(image):
    if image is None:
        return "Please Upload a valid sky image!"
    pred_cloud_coverage = predict_cloud_coverage(image, clip_model, cbt_model)
    if pred_cloud_coverage <= 33.0:
        s = "There is Low Cloud Coverage!   Predicted Opaque Cloud Coverage: {}%".format(
            pred_cloud_coverage)
    elif pred_cloud_coverage > 33.0 and pred_cloud_coverage <= 66.0:
        s = "There is Moderate Cloud Coverage!   Predicted Opaque Cloud Coverage: {}%".format(
            pred_cloud_coverage)
    else:
        s = "There is High Cloud Coverage!   Predicted Opaque Cloud Coverage: {}%".format(
            pred_cloud_coverage)
    return s
 
 
# Create the Gradio app
iface = gr.Interface(
    fn=predict,
    inputs=[Image(label="Upload a Sky Cam image")],
    outputs=[Textbox(label="Prediction")],
    title="GFG EcoTech Hackathon: Cloud Coverage Calculator From a Sky Cam Image",
    description='Upload only a skycam image and get the opaque cloud coverage in %     |     (Low: 0-33 | Moderate: 33-66 | High: 66-100)     |     <a href="https://drive.google.com/drive/folders/1r8mTWEG4XEBZDg0TNyXTYkGzZVixXvcj?usp=drive_link">Find Sample Testing Images Here!</a>',
)
 
# Run the Gradio app
iface.launch(debug=True)

Output:

Running on local URL:  http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.

The output will look like below. We can uplad the images by clicling “Click to Upload”

Predictions Window

Predictions

Cloud Coverage Prediction using Skycam Images

Cloud coverage prediction is critical in weather forecasting and a variety of applications such as solar energy generation, aviation, and climate monitoring. Accurate forecasts help decision-makers and sectors plan for and adapt to changing weather conditions. The advancement of artificial intelligence and computer vision techniques in recent years has created new opportunities for enhancing cloud coverage forecasts.

One promising approach is the use of SkyCam images.

In the face of rapidly changing global climate patterns, there is an urgent need for innovative tools and technologies to better understand and predict weather-related phenomena.
One crucial aspect of climate analysis is the assessment of cloud coverage, which plays a pivotal role in influencing weather conditions and climate trends.
Experts may not always be available to monitor climatic shifts. Therefore, developing an automated weather monitoring system is crucial for various applications, including agriculture and disaster management.

The purpose of this research is to estimate the opaque Cloud Coverage from a Skycam Image using AI/ML methodologies.

Cloud Coverage Prediction using Skycam Images

Table of Content

Cloud Coverage Prediction using SkyCam Images
Implementations Cloud Coverage Prediction using SkyCam Images
Cloud Coverage Prediction Models:
Part I. Model Building & Traning Pipeline
A. Clip Model Finetuning
B. Catboost Regressor Model Building
Part II. UI Inference Codes for Deployed Model
Results:

Part II. UI Inference Codes for Deployed Model

A. cloud_coverage_pipeline.py code:

Python3

app.py code:

Python3

Predictions

Cloud Coverage Prediction using Skycam Images

Table of Content

Similar Reads

Contact Us