Most know data science as training as a machine learning model. In a real project, the model is just a small component of the work. Success or failure hinges on what comes before and what comes after it. There are always six steps, and they are always performed in the same order in a proper project. Let’s go through them individually.
1. CORE COMPETENCE: Understanding business question
Don’t open the dataset first. First, ask what the decision will be about this project. Improving our sales is not a question, it’s a wish. Which customers are likely to stop buying next month is a question because you can measure the answer and do something about it. Record that question in one line and continue to think about it throughout the process. At this one crucial step, half of the failed projects fail.
2. COLLECTING THE DATA
You go and fetch that data from where it is. Company databases via SQL queries, third party APIs, exported CSV files and in some cases via web scraping. The key is to correctly join these sources. If you do one of the joins wrong, such as the order number is joined to the wrong customer ID, then all the numbers that follow this step will be quietly incorrect. Avoiding the pain of days to come, always check the number of rows before and after a join.
3. CLEANING THE DATA
It is the longest step, and no one wants to take it. Missing values, duplicate rows, incorrect date formats and city spelled four ways. The survey uncovered that data professionals spend approximately 45% of time just on data preparation. Most of this work is done by pandas and NumPy, but it is up to you! Cleaning is dull but it’s what makes all the subsequent results reliable.
4. EXPLORING AND VISUALISING
Always have a look at the data before touching any model. Use Matplotlib or Seaborn to make plots of each distribution, test for outliers and compare columns. This step is referred to as Exploratory Data Analysis (EDA). Very often the solution is just here and there is no need for a model at all. It also prevents you from having a large class on a trivial problem.
5. BUILDING AND TRAINING THE MODEL
The model is built and trained by following the following steps:
Now we come to the bit everyone mentions. Divide a dataset into training and test sets, select an algorithm from Scikit-learn, and learn the algorithm. Don’t test on the same data that it was trained on; it will tell you that it is 99% accurate on that test set, and it will fail on real data! This error is referred to as overfitting. Experiment with 2 or 3 algorithms and compare them appropriately rather than assuming the first one is right.
6. DEPLOYING AND MONITORING
A model that is confined in a jupyter notebook doesn’t help anyone. Publish it to use by real people, using FastAPI, Flask, or this handy Streamlit app. Just watch it post-launch as well. A model that was correct last year starts to become incorrect over time. This is known as model drift and the only solution is retraining. Meaning, deployment is not the end of the road.