Introduction

Binary logistic regression is a popular machine learning approach for binary classification tasks. One or more predictor variables are used to predict the likelihood of a binary result in this regression study. Although named “regression,” logistic regression is mostly used for categorization. This article covers binary logistic regression’s concepts, implementation, Advantages and Disadvantages of Binary Logistic Regression

What is a Binary Logistic Regression?

Machine learning uses Binary Logistic Regression for binary classification. It predicts whether an input belongs to one of two classes (typically 0 or 1, true or false). Logistic regression models the relationship between input data and target class probability using the logistic (sigmoid) function, unlike linear regression, which predicts a continuous value.

The logistic function outputs a value between 0 and 1 that represents the input’s positive class probability (typically 1). Logistic regression is appropriate for predicted customer turnover, disease diagnosis, and email spam classification, which have just two outcomes.

Maximum Likelihood Estimation is used to identify the best-fitting parameters (weights) that minimize the error between predicted probability and actual outcomes in logistic regression. In models with multiple characteristics, regularization can prevent overfitting.

Binary logistic regression is straightforward and interpretable, but it assumes a linear relationship between features and target class log-odds. It is extensively used since it is efficient and effective in many binary classification jobs.

Logistic regression Model

The logistic regression model seeks a link between characteristics and the binary dependent variable. Logistic regression predicts the probability of the outcome falling into one of two classes: 1 for a good event and 0 for a negative event.

Linear regression predicts continuous values; logistic regression predicts probability. These probabilities indicate the possibility that an input is affirmative. In medical diagnosis, logistic regression can estimate a patient’s disease probability based on their medical history.

How Does Logistic Regression Work?

Logistic regression works by transforming a linear combination of input features into a log-odds value and then applying the logistic (sigmoid) function to convert it into a probability.

Log-Odds Transformation: Logistic regression first linearizes input features. A linear combination with two input characteristics can look like:

𝑧 = 𝑤₀ + 𝑤₁ 𝑥₁ + 𝑤₂ 𝑥₂
​

where:

• Input features (e.g., age, income) are represented by 𝑥₁ and 𝑥₂.
• 𝑤_1​,𝑤₂,𝑤₃ are the model coefficients (weights) that need to be learned during the training process.

Logistic (Sigmoid) Transformation: After calculating log-odds, the sigmoid function converts it to a probability. The sigmoid function’s S-shaped curve squashes real numbers between 0 and 1. This makes the output consistent with probability.

This change makes logistic regression appropriate for binary classification by modeling probabilities.

Training Logistic Regression Models

Training a logistic regression model is to find weights that minimize the gap between predicted probability and binary outcomes. Using Maximum Likelihood Estimation, the likelihood function is maximized.

• Maximum Likelihood Estimation (MLE): Maximum Likelihood Estimation (MLE) in logistic regression calculates the probability of receiving observed data based on model parameters. The goal is to discover the parameters that maximize the chance of the given outcomes.

To simplify mathematics and computation, it is more customary to minimize the log-likelihood (the logarithm of the likelihood function) rather than the likelihood. Log-likelihood functions for logistic regression summarise discrepancies between predicted probability and binary labels.

• Optimization: To identify the best-fitting parameters, optimization techniques such as gradient descent are often used. Gradient descent iteratively adjusts parameters to lower log-likelihood, finding optimal weights.

Regularization in Logistic Regression

Logistics regression works effectively in many circumstances, however overfitting occurs when there are too many characteristics or noisy data. If the model matches training data too well, it captures noise or random fluctuations that don’t generalize to unseen data.

For this, logistic regression is commonly regularized. Regularization adds a penalty term to the loss function to discourage complex models with high weights. This prevents overfitting and makes the model less sensitive to tiny training data changes.

Logistic regression uses two regularization methods:

• L1 Regularization (Lasso): This penalises coefficient absolute value proportionally. Some coefficients are driven to zero by L1 regularization, removing less significant features from the model.
• L2 Regularization (Ridge): This approach penalizes coefficient squares. L2 regularization reduces coefficients but does not zero them. Noise or irrelevant features can cause huge coefficients, but it works.

Regularization is regulated by a penalty strength hyperparameter. We can balance data fitting with model simplicity by setting this hyperparameter.

Model Evaluation Metrics

The logistic regression model must be tested after training. Many metrics are used to evaluate binary classification models. Popular metrics include:

1. Accuracy: The percentage of correct predictions (true positives and negatives) is accuracy. This metric is often used, although it might be misleading when the data is imbalanced (one class is considerably more frequent than the other).

Accuracy= (TP+TN) / (FP+FN+TP+TN​)

2. Precision and Recall: In imbalanced datasets, precision and recall provide a more complete knowledge of model performance.

• Precision measures how frequently positive predictions are true. It’s useful when false positives cost a lot.

Precision= TP / (FP+TP)​

Recall measures how many positives the model detected correctly. It helps when false negatives are expensive.

Recall= TP / (FN+TP)​

3. F1-Score: The harmonic mean of precision and recall is F1. It helps balance precision and recall in uneven datasets.

F1-Score= 2×(Precision×Recall) / (Precision+Recall​)

4. Area Under the ROC Curve (AUC-ROC): ROC curve compares true positive rate (recall) against false positive rate. The AUC (Area Under the Curve) measures the classifier’s ability to differentiate positive and negative classes. An AUC of 1 represents perfect categorization, whereas 0.5 indicates random guessing.

Advantages of Binary Logistic Regression

• Simplicity and Interpretability: Implementing and understanding logistic regression is simple. Coefficients that can be read as target variable log-odds make the model clear and explainable.
• Efficiency: Logistic regression is computationally efficient, especially for small datasets with few characteristics. It uses fewer resources than neural networks or SVMs.
• Probabilistic Interpretation: Logistic regression produces probabilities, which provide class predictions and confidence. Many applications, such as medical diagnostics and risk assessment, need understanding outcome likelihood.
• Regularization Support: Logistic regression supports L1 and L2 regularization methods, which reduce overfitting and increase model generalization.

Binary Logistic Regression Disadvantages

• Linear Decision Boundaries: Logistic regression presupposes a linear relationship between input features and output log-odds. It is less suitable for non-linearly separable class problems. Complex models like decision trees, random forests, and neural networks may perform better.
• Sensitivity to Outliers: Logistic regression can be sensitive to data outliers. Outliers can distort or confuse forecasts by disproportionately affecting estimated coefficients.
• Assumes Feature Independence: Logistic regression implies input features are independent. High feature correlation might make a model unstable or perform poorly.
• Requires Large Sample Sizes for Stable Estimates: Logistic regression requires large sample sizes for stable estimates. Small or imbalanced datasets may not work effectively. Penalized regression or ensemble approaches may work better.

Applications of Logistic Regression

Logistic regression is used in many disciplines for binary categorization. Common uses include:

• Medical Diagnostics: A person’s medical records can be used to predict whether they have cancer.
• Customer Churn Prediction: Using usage patterns and demographic data to predict customer churn from a subscription service.
• Credit Scoring: Predicting loan default based on financial history and other criteria.
• Marketing: Predicting customer response to marketing campaigns to better target ads.

Binary Logistic Regression in python

Here’s an example using Python’s scikit-learn library to implement logistic regression:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# Example dataset (X: features, y: target variable)
X = np.array([[2, 3], [1, 2], [4, 5], [6, 7], [7, 8], [8, 9]])
y = np.array([0, 0, 1, 1, 1, 1])

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Standardize the features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Initialize and train the logistic regression model
model = LogisticRegression()
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate the model
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")
print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
print("Classification Report:")
print(classification_report(y_test, y_pred))

Conclusion

Machine learning tool binary logistic regression is powerful and versatile for binary classification applications. Its simplicity, interpretability, and probabilistic output make it appealing for many applications. Machine learning practitioners can use logistic regression for real-world problems by knowing the model’s workings, training method, assessment measures, and constraints. It is one of the most popular predictive modeling and classification algorithms, despite its assumptions about linearity between characteristics and outcomes.