Does a More Complex ML Model Actually Give Better Results?

I recently compared Logistic Regression and XGBoost on a customer churn problem.

I expected XGBoost to do better, but the results were actually the opposite. Logistic Regression performed better across all the metrics I compared.

I also ran 5-fold cross-validation to make sure this wasn't just because of one data split. The results were similar there too.

So, I thought this was worth sharing.

How I compared them

I used the Telco Customer Churn dataset, which has 7,043 customer records. Around 26.5% of them had churned. I kept the setup the same for both models so the comparison was fair:

• 80% data for training and 20% for testing

• Same train/test split

• Numerical features were scaled

• Categorical features were one-hot encoded

• Compared using Accuracy, Precision, Recall, F1 and ROC-AUC

I first ran Logistic Regression and then XGBoost on the same setup.

The first results

Logistic Regression performed better across all five metrics.

But this was only one train/test split. I didn't want to conclude anything based on that alone, so I checked both models again using 5-fold cross-validation.

Checking with cross-validation

I ran 5-fold cross-validation for both models using the same evaluation metrics. This gave me a better view of how the models performed across different splits of the data.

The same pattern showed up again. Logistic Regression stayed ahead across all five metrics.

So the first result wasn't just coming from one particular data split.

What I took from this

This doesn't mean Logistic Regression is better than XGBoost.

I used XGBoost without much tuning, and with a different dataset or setup, the results could easily change.

For me, the main takeaway was simple: start with a good baseline before adding complexity.

A more complex model should earn its place by giving you something better performance, better handling of the problem, or some other clear advantage.

In this case, it didn't.

Experiment

I’ve kept the complete experiment, code, results, and cross-validation in GitHub if you want to take a closer look:

GitHub: Experiment