Three Years of Creating ML Projects : What Did I Learn?

https://medium.com/@priyankatariya.26/three-years-of-creating-ml-projects-what-did-i-learn-704dd4b45c22

The first ever project I worked upon when I was just getting started with my journey in AI three years ago was a diabetes prediction model using various supervised learning models (KNN, SVM, Logistic Regression,Random Forest, Decision Tree, etc) for comparative analysis of different learning mechanisms. I took the first dataset I found off of Kaggle, downloaded it, panda-ed it onto my anaconda environment and started training. For weeks after building this project using half scraped code I had found in my professor’s PPTs and google, I still did not fully understand what the point of this whole concept was.

Why did I just train this random dataset formatted in csv?

Why is the training to testing split typically kept at a 8:2 ratio?

Why are there so many libraries for ML and which one works optimally for what function?

What do I even do with this trained model?

All of these questions bugged me until I googled ‘how to deploy a trained ML model?’

This one simple question, made me hit the mother lode.

Quite Literally.

I came across a plethora of insane applications where Machine Learning could be used AND deployed for real world scenarios that held a lot of significance. It made me realise that even a simple user input system that takes into consideration certain values such as a person’s RBC count, cellular diameter, blood group, family history, etc could predict if that individual has diabetes or not within a matter of seconds. To make it better, the model could also be trained to predict the type of diabetes the patient has.

Inspired by this discovery and incredibly proud of myself for having replicated a project that was already existing and well-known, I set out on a journey to fully understand what exactly ML stood for. What it could create and to what extent it could learn.

Just to give a gist of the different types of approaches based on different types of data available, Machine Learning can be simply summarised into three main categories. Namely; Supervised Learning, Unsupervised Learning and Reinforcement Learning.

Classification of ML Types

Supervised Learning works on already existing data with labelled outputs. The model recognises trends and patterns in the data that lead to the outputs desired by the program. It then tests this knowledge on new unseen data formatted in the same way the training data was set and tries to calculate how well it predicted the output.

Unsupervised Learning works with data that does not have labeled outputs. The model tries to find hidden patterns, structures, or relationships in the data without knowing the correct answers ahead of time. It groups or clusters similar data points together or reduces data dimensions to highlight essential features. Once trained, the model can apply this knowledge to new, unseen data to organise or interpret it based on the patterns it discovered.

Reinforcement Learning (RL) involves training a model through interactions with an environment. The model learns by performing actions and receiving feedback in the form of rewards or penalties. The goal is to maximise the cumulative reward over time by learning which actions lead to the best outcomes. The model improves as it tests new strategies, adapting its behaviour to optimise long-term success based on past experience.

Once I gained a thorough understanding of these 3 modules, I felt unstoppable. I explored domains that were not worked upon much and I tried to optimise already existing projects to automate them. I created a cyberbullying detection system, a facial emotion recognition model, a speech processing based confidence classification model that could detect how confident you are during an interview or viva, a customer churn model for a fintech, a Gen-AI based blog generator and much more.

I set my foot into freelancing where I created a ML and DL based mood prediction model for students in universities depending on their financial background, familial support, mental health and many more factors that finally would suggest ways in which a student could be helped in his or her education, personal life or financial well-being. Another really interesting project I worked upon dealt with images and videos that had to be reconstructed and enhanced in quality. I tackled this through a hybrid SR-GANs (Super Resolution — Generative Adversarial Networks) model.

Although there is a lot of unexplored space that I one day wish to master, these past three years have made me grow and learn unexpectedly well as an AI engineer who is soon going to be launched into the corporate world to develop impactful (hopefully) AI/ML models.

Here is one of my now available research paper while the rest are still under the publication process:

→ https://ieeexplore.ieee.org/abstract/document/10616957/?casa_token=EcqnU9NvqfQAAAAA:4P6_A4T8rRQmkpzktLNp8g0lnPAnEpTvVMkDDDIjBqLW9avyDmoXf7ob7Q3rViJphy49X56JbIk