Fraud Detection and Insights Using Machine Learning
Table of Contents¶
- Introduction and Motivation
- Data explanation
- Data cleaning and processing
- Exploratory Data Analysis
- Models, algorithms ,and training
- Interpretation of Results
- Conclusion
- References
Introduction¶
Online financial payments have transformed the way people manage their money, offering convenience, speed, and accessibility like never before. However, this rise in digital transactions has also led to an increase in fraudulent activities, posing serious risks to individuals, businesses, and financial institutions. Fraud detection for online financial payments plays a vital role in addressing these challenges by ensuring the safety and security of digital financial systems.
Motivation¶
One of the main reasons for choosing this topic is the rapid adoption of digital payment methods, such as mobile wallets, online banking, and cryptocurrencies. With the growing number of users, fraudsters have developed more advanced techniques, making it increasingly difficult for traditional methods to detect and prevent fraud effectively. This calls for innovative approaches to stay ahead of these threats. The alarming rise in payment fraud incidents highlights the critical need for robust fraud detection mechanisms. In 2023 alone, consumers globally lost a staggering $10 billion to fraud, marking a 14% increase from the previous year. This trend underscores the urgent necessity for financial institutions and businesses to implement advanced fraud detection solutions to protect their customers and themselves from financial losses and reputational damage.
Another motivation is the financial and reputational damage caused by fraud. Fraud not only results in monetary losses but also erodes trust in online platforms and creates significant challenges for businesses trying to recover. Developing effective fraud detection systems can help protect users and businesses while fostering confidence in digital payment systems.
Additionally, this topic allows the exploration of cutting-edge technologies like machine learning, blockchain, and advanced data analysis to build better fraud detection systems. These technologies can improve accuracy, protect user privacy, and comply with important regulations, ensuring secure and trustworthy financial transactions.
Finally, studying fraud detection provides valuable insights into fraudulent behavior and system vulnerabilities. This knowledge can help create adaptable systems that respond effectively to emerging threats, enhancing the overall security of online payments.
Important¶
- Each row represents a Single Transaction
- Ignore the warning messages in a few code cells
Important Libraries and Modules¶
This project utilizes various Libraries and Modules majority of them have been listed and imported below, with a few uncommon modules such as lighbgm being installed on the go, and some of the libraries that are specific to some problems have been installed later along with the code
In [ ]:
!pip install lightgbm
!pip install catboost
!pip install shap
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
Data Explanation¶
This is an Online payment fraud dataset with total rows = 6362620 and total columns= 16. column description
- step: Represents a unit of time where 1 step equals 1 hour.
- type: Type of online transaction.
- amount: The amount of the transaction.
- nameOrig: Customer starting the transaction.
- oldbalanceOrg: Balance before the transaction.
- newbalanceOrig: Balance after the transaction.
- nameDest: Recipient of the transaction.
- oldbalanceDest: Initial balance of recipient before the transaction.
- newbalanceDest: The new balance of the recipient after the transaction.
- isFraud: Indicates whether the transaction is fraudulent.
- isFlaggedFraud: Indicates whether the transaction is marked as Fraud by the existing system.
- State: The U.S. state where the transaction originated.
- merchant_category: The category of the merchant to which the payment was made.
- time_of_day: Specifies whether the transaction occurred in the morning, afternoon, evening, or night.
- Currency: All transactions are in USD.
- time: Timestamp of the transaction.
Important
- This is a Highly imbalanced, unclean dataset replicating real-life scenarios during Fraud detection at Financial Companies. The imbalance is of about less than 1%, indicating less than 1% data representing Fraudulent Transaction
- With the help of States, Time, Merchant category, etc, we can find out a lot about the current trends in Fraud across various segments
In [30]:
file_path = "Fraud Dataset.csv"
data = pd.read_csv(file_path)
In [31]:
data.head()
Out[31]:
| step | type | amount | nameOrig | oldbalanceOrg | newbalanceOrig | nameDest | oldbalanceDest | newbalanceDest | isFraud | isFlaggedFraud | State | merchant_category | time_of_day | Currency | time | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | NaN | PAYMENT | 9839.64 | C1231006815 | 170136.0 | 160296.36 | M1979787155 | 0.0 | 0.0 | 0 | 0.0 | Alaska | Groceries | Evening | USD | 20:09:09 |
| 1 | 1.0 | PAYMENT | 1864.28 | C1666544295 | NaN | 19384.72 | M2044282225 | 0.0 | 0.0 | 0 | 0.0 | Kentucky | Housing | Night | USD | NaN |
| 2 | 1.0 | TRANSFER | 181.00 | C1305486145 | 181.0 | 0.00 | C553264065 | 0.0 | 0.0 | 1 | 0.0 | Washington | Housing | Night | USD | 22:38:16 |
| 3 | 1.0 | CASH_OUT | 181.00 | C840083671 | 181.0 | 0.00 | C38997010 | 21182.0 | 0.0 | 1 | 0.0 | Maryland | Groceries | Night | USD | 23:31:48 |
| 4 | NaN | PAYMENT | 11668.14 | C2048537720 | 41554.0 | 29885.86 | NaN | 0.0 | 0.0 | 0 | 0.0 | Indiana | NaN | Evening | USD | 21:00:45 |
All the datatypes are assigned correctly upon initial inspection
In [34]:
data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 6362620 entries, 0 to 6362619 Data columns (total 16 columns): # Column Dtype --- ------ ----- 0 step float64 1 type object 2 amount float64 3 nameOrig object 4 oldbalanceOrg float64 5 newbalanceOrig float64 6 nameDest object 7 oldbalanceDest float64 8 newbalanceDest float64 9 isFraud int64 10 isFlaggedFraud float64 11 State object 12 merchant_category object 13 time_of_day object 14 Currency object 15 time object dtypes: float64(7), int64(1), object(8) memory usage: 776.7+ MB
Null Values¶
We do have about 600k missing values for each column except the isFraud Column which makes it about 10% null values with respect to each column. While we do have a huge amount of data already, deleting these values seems a good idea, but we need to investigate further whether or not we can directly delete or we can impute some values
In [41]:
print("Missing Values Summary:")
data.isnull().sum()
Missing Values Summary:
Out[41]:
step 636931 type 636971 amount 635503 nameOrig 635400 oldbalanceOrg 636465 newbalanceOrig 636671 nameDest 634600 oldbalanceDest 636399 newbalanceDest 635336 isFraud 0 isFlaggedFraud 634204 State 635548 merchant_category 635844 time_of_day 634790 Currency 635379 time 634874 dtype: int64
There are No Duplicate rows recorded
In [36]:
#Check for duplicate rows
duplicates = data.duplicated().sum()
print(f"Number of duplicate rows: {duplicates}")
print("\n")
Number of duplicate rows: 0
This gives an overview of Descriptive Statistics within the Numerical columns, showcasing mean, count, std, min, max, and quartiles for each column, which can be used for further analysis for example, what is the mean amount of txn in USD across the entire dataset?? or what is the STD for a particular column??
In [40]:
#Check basic statistics of numerical columns
data.describe()
Descriptive Statistics:
Out[40]:
| step | amount | oldbalanceOrg | newbalanceOrig | oldbalanceDest | newbalanceDest | isFraud | isFlaggedFraud | |
|---|---|---|---|---|---|---|---|---|
| count | 5.725689e+06 | 5.727117e+06 | 5.726155e+06 | 5.725949e+06 | 5.726221e+06 | 5.727284e+06 | 6.362620e+06 | 5.728416e+06 |
| mean | 2.434369e+02 | 1.801073e+05 | 8.338695e+05 | 8.553543e+05 | 1.101267e+06 | 1.224922e+06 | 1.290820e-03 | 2.793093e-06 |
| std | 1.423789e+02 | 6.044018e+05 | 2.888457e+06 | 2.924352e+06 | 3.406300e+06 | 3.676676e+06 | 3.590480e-02 | 1.671253e-03 |
| min | 1.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 |
| 25% | 1.560000e+02 | 1.338853e+04 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 |
| 50% | 2.390000e+02 | 7.491213e+04 | 1.422300e+04 | 0.000000e+00 | 1.327682e+05 | 2.146901e+05 | 0.000000e+00 | 0.000000e+00 |
| 75% | 3.350000e+02 | 2.087943e+05 | 1.073840e+05 | 1.442456e+05 | 9.434232e+05 | 1.112120e+06 | 0.000000e+00 | 0.000000e+00 |
| max | 7.430000e+02 | 9.244552e+07 | 5.958504e+07 | 4.958504e+07 | 3.560159e+08 | 3.561793e+08 | 1.000000e+00 | 1.000000e+00 |
In [ ]:
## Data Explantion
In [39]:
#Summary of unique values in each column
print("Unique Values in Each Column:")
for column in data.columns:
print(f"{column}: {data[column].nunique()} unique values")
Unique Values in Each Column: step: 743 unique values type: 5 unique values amount: 4856525 unique values nameOrig: 5719701 unique values oldbalanceOrg: 1680912 unique values newbalanceOrig: 2419939 unique values nameDest: 2495936 unique values oldbalanceDest: 3256605 unique values newbalanceDest: 3226084 unique values isFraud: 2 unique values isFlaggedFraud: 2 unique values State: 50 unique values merchant_category: 8 unique values time_of_day: 4 unique values Currency: 1 unique values time: 82742 unique values
In [47]:
# Critical columns: 'step', 'type', 'amount', 'isFraud'
data_cleaned = data.dropna(subset=['step', 'type', 'amount', 'isFraud']).copy()
# Impute missing numerical values with the median
numerical_columns = ['oldbalanceOrg', 'newbalanceOrig', 'oldbalanceDest', 'newbalanceDest']
for col in numerical_columns:
data_cleaned.loc[:, col] = data_cleaned[col].fillna(data_cleaned[col].median())
#Impute missing categorical values with the mode
categorical_columns = ['State', 'merchant_category', 'time_of_day']
for col in categorical_columns:
if col in data_cleaned.columns: # Ensure the column exists in the dataset
data_cleaned.loc[:, col] = data_cleaned[col].fillna(data_cleaned[col].mode()[0])
# Drop low-variance or irrelevant columns
irrelevant_columns = ['Currency', 'nameOrig', 'nameDest', 'time']
data_cleaned = data_cleaned.drop(columns=irrelevant_columns, errors='ignore')
#Reset index after cleaning
data_cleaned = data_cleaned.reset_index(drop=True)
#Output cleaned dataset summary
print("Cleaned Dataset Summary:")
print(data_cleaned.info())
print("\nMissing Values after Cleaning:")
print(data_cleaned.isnull().sum())
print("\nFirst few rows of the cleaned dataset:")
print(data_cleaned.head())
Cleaned Dataset Summary: <class 'pandas.core.frame.DataFrame'> RangeIndex: 4637838 entries, 0 to 4637837 Data columns (total 12 columns): # Column Dtype --- ------ ----- 0 step float64 1 type object 2 amount float64 3 oldbalanceOrg float64 4 newbalanceOrig float64 5 oldbalanceDest float64 6 newbalanceDest float64 7 isFraud int64 8 isFlaggedFraud float64 9 State object 10 merchant_category object 11 time_of_day object dtypes: float64(7), int64(1), object(4) memory usage: 424.6+ MB None Missing Values after Cleaning: step 0 type 0 amount 0 oldbalanceOrg 0 newbalanceOrig 0 oldbalanceDest 0 newbalanceDest 0 isFraud 0 isFlaggedFraud 462132 State 0 merchant_category 0 time_of_day 0 dtype: int64 First few rows of the cleaned dataset: step type amount oldbalanceOrg newbalanceOrig oldbalanceDest \ 0 1.0 PAYMENT 1864.28 14267.73 19384.72 0.0 1 1.0 TRANSFER 181.00 181.00 0.00 0.0 2 1.0 CASH_OUT 181.00 181.00 0.00 21182.0 3 1.0 PAYMENT 7817.71 53860.00 46042.29 0.0 4 1.0 PAYMENT 7107.77 183195.00 176087.23 0.0 newbalanceDest isFraud isFlaggedFraud State merchant_category \ 0 0.0 0 0.0 Kentucky Housing 1 0.0 1 0.0 Washington Housing 2 0.0 1 0.0 Maryland Groceries 3 0.0 0 0.0 Massachusetts Entertainment 4 0.0 0 0.0 Wyoming Groceries time_of_day 0 Night 1 Night 2 Night 3 Evening 4 Evening
Data Cleaning and Preprocessing¶
Missing Data¶
- After careful analysis, we decide to drop the Null values with respect to columns 'step', 'type', 'amount', 'isFraud'.
- For columns 'oldbalanceOrg', 'newbalanceOrig', 'oldbalanceDest', and 'newbalanceDest', as they are numerical, we impute the missing values with the median of those columns.
- For Columns 'State', 'merchant_category', and 'time_of_day', as they have Strings or categorical values, we impute them using Mode
- Thus, Missing Data is taken care off
Irrelevant Columns¶
Columns such as 'Currency', 'nameOrig', 'nameDest', and 'time' have been deleted because Currency is USD across every row, and time is not relevant because we already have another column stating whether it was morning, evening, afternoon or night hence duplicated data is not necessary
Dropped all the Irrelevant Columns
Further preprocessing Like encoding and handling imbalance, Feature Engineering can be seen before model Building
In [5]:
# Impute missing values in 'isFlaggedFraud' with 0 (non-flagged assumption)
data_cleaned['isFlaggedFraud'] = data_cleaned['isFlaggedFraud'].fillna(0)
# Verify if all missing values are handled
print("Missing Values after Handling 'isFlaggedFraud':")
print(data_cleaned.isnull().sum())
Missing Values after Handling 'isFlaggedFraud': step 0 type 0 amount 0 oldbalanceOrg 0 newbalanceOrig 0 oldbalanceDest 0 newbalanceDest 0 isFraud 0 isFlaggedFraud 0 State 0 merchant_category 0 time_of_day 0 dtype: int64
In [7]:
# Create a count plot for the 'isFraud' column
plt.figure(figsize=(8, 5))
sns.countplot(x='isFraud', data=data_cleaned, palette="viridis")
# Use a logarithmic scale for the y-axis
plt.yscale('log')
# Annotate the bars with counts
for index, value in enumerate(data_cleaned['isFraud'].value_counts()):
plt.text(index, value, f'{value:,}', ha='center', va='bottom', fontsize=12)
# Add labels and title
plt.title("Count Plot for 'isFraud' Column (Log Scale)", fontsize=16)
plt.xlabel("isFraud (0 = Non-Fraud, 1 = Fraud)", fontsize=14)
plt.ylabel("Count (Log Scale)", fontsize=14)
plt.grid(axis='y', alpha=0.3)
plt.show()
/var/folders/_6/0lw9xh_90gl_6d2dc41byfzh0000gn/T/ipykernel_3138/1314605840.py:6: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.countplot(x='isFraud', data=data_cleaned, palette="viridis")
Exploratory Data Analysis¶
Count Plot for the target variable isFraud
-Class Imbalance: The graph shows the distribution of non-fraud (isFraud = 0) and fraud (isFraud = 1) transactions. A large difference between the heights of the bars indicates a highly imbalanced dataset, which is typical in fraud detection scenarios.
-Logarithmic Scale: The y-axis uses a logarithmic scale to make the smaller fraud counts (isFraud = 1) visible while maintaining the proportional difference compared to non-fraud transactions.
-Fraud Prevalence: The graph visually emphasizes how rare fraudulent transactions are compared to non-fraudulent ones, which helps highlight the challenge of detecting fraud in such datasets.
In [8]:
data_cleaned
Out[8]:
| step | type | amount | oldbalanceOrg | newbalanceOrig | oldbalanceDest | newbalanceDest | isFraud | isFlaggedFraud | State | merchant_category | time_of_day | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1.0 | PAYMENT | 1864.28 | 14267.73 | 19384.72 | 0.00 | 0.00 | 0 | 0.0 | Kentucky | Housing | Night |
| 1 | 1.0 | TRANSFER | 181.00 | 181.00 | 0.00 | 0.00 | 0.00 | 1 | 0.0 | Washington | Housing | Night |
| 2 | 1.0 | CASH_OUT | 181.00 | 181.00 | 0.00 | 21182.00 | 0.00 | 1 | 0.0 | Maryland | Groceries | Night |
| 3 | 1.0 | PAYMENT | 7817.71 | 53860.00 | 46042.29 | 0.00 | 0.00 | 0 | 0.0 | Massachusetts | Entertainment | Evening |
| 4 | 1.0 | PAYMENT | 7107.77 | 183195.00 | 176087.23 | 0.00 | 0.00 | 0 | 0.0 | Wyoming | Groceries | Evening |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 4637833 | 743.0 | CASH_OUT | 339682.13 | 339682.13 | 0.00 | 0.00 | 339682.13 | 1 | 0.0 | Illinois | Groceries | Afternoon |
| 4637834 | 743.0 | TRANSFER | 6311409.28 | 6311409.28 | 0.00 | 0.00 | 0.00 | 1 | 0.0 | New Jersey | Entertainment | Afternoon |
| 4637835 | 743.0 | CASH_OUT | 6311409.28 | 6311409.28 | 0.00 | 68488.84 | 6379898.11 | 1 | 0.0 | Tennessee | Entertainment | Night |
| 4637836 | 743.0 | TRANSFER | 850002.52 | 850002.52 | 0.00 | 0.00 | 0.00 | 1 | 0.0 | North Carolina | Housing | Evening |
| 4637837 | 743.0 | CASH_OUT | 850002.52 | 850002.52 | 0.00 | 6510099.11 | 7360101.63 | 1 | 0.0 | Maryland | Entertainment | Night |
4637836 rows × 12 columns
In [9]:
# Filter the data for fraud transactions only
fraud_data = data_cleaned[data_cleaned['isFraud'] == 1]
# Calculate the average transaction amount grouped by 'time_of_day'
average_txn_fraud = fraud_data.groupby(['time_of_day'])['amount'].count().reset_index()
# Create the bar plot
plt.figure(figsize=(10, 6))
sns.barplot(
x='time_of_day',
y='amount',
data=average_txn_fraud,
palette='viridis'
)
# Add labels and title
plt.title("Average Transaction Amount by Time of Day (Fraud Only)", fontsize=16)
plt.xlabel("Time of Day", fontsize=14)
plt.ylabel("Average Transaction Amount", fontsize=14)
plt.grid(axis='y', alpha=0.3)
plt.show()
/var/folders/_6/0lw9xh_90gl_6d2dc41byfzh0000gn/T/ipykernel_3138/2903006207.py:12: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.barplot(
Bar plot for Average Transactions vs Time_Of_Day¶
Average Fraud Transaction Amount by Time of Day: The graph shows the average transaction amount for fraudulent transactions grouped by different periods of the day: Morning, Afternoon, Evening, and Night.
Distribution of Fraudulent Transaction Values: The bar heights indicate whether certain times of day (e.g., Night) tend to have higher-value fraudulent transactions compared to others.
Identifying Fraud Patterns: If the graph shows higher average transaction amounts during specific times, that is, Night, it might indicate when fraudsters are most active or successful.
Risk Assessment: Understanding the time-of-day patterns can help assess when transactions are riskier, allowing businesses to strengthen fraud detection measures during high-risk periods.
In [10]:
# Group the data for fraud transactions by city and calculate the total transaction amount
fraud_city_totals = data_cleaned[data_cleaned['isFraud'] == 1].groupby('State')['amount'].sum().reset_index()
# Sort by total transaction amount in descending order
fraud_city_totals = fraud_city_totals.sort_values(by='amount', ascending=False)
# Create the bar plot
plt.figure(figsize=(12, 8))
sns.barplot(
y='State',
x='amount',
data=fraud_city_totals,
palette='viridis'
)
# Add labels and title
plt.title("Total Transaction Amount by City (Fraud Only)", fontsize=16)
plt.xlabel("Total Transaction Amount", fontsize=14)
plt.ylabel("City", fontsize=14)
plt.grid(axis='x', alpha=0.3)
plt.show()
/var/folders/_6/0lw9xh_90gl_6d2dc41byfzh0000gn/T/ipykernel_3138/1437951722.py:9: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(
Sum of Transactions that are Fraud segrated with respect to States in the USA¶
Total Fraudulent Transaction Amount by State:
- The graph displays the total monetary value of fraudulent transactions (
isFraud = 1) for each state. - States are represented along the y-axis, while the x-axis shows the total transaction amount.
- The graph displays the total monetary value of fraudulent transactions (
Sorting by Total Amount:
- The states are sorted in descending order of the total fraudulent transaction amount.
- This ensures that the states with the highest fraud-related monetary losses are displayed at the top.
Insights¶
High-Risk States:
- States like New York, Texas, California, and Florida dominate the total fraudulent transaction amounts. These states experience significantly higher financial losses due to fraud, suggesting they are hotspots for fraudulent activity.
Moderate-Risk States:
- States such as Massachusetts, Pennsylvania, Illinois, and Georgia show considerable fraudulent transaction amounts but are less affected than the top-tier states.
Low-Risk States:
- States like Wyoming, Alaska, and Hawaii exhibit the least amount of fraudulent transaction activity. These states face minimal financial impact due to fraud.
In [11]:
# Calculate the total transaction amount for fraud and non-fraud grouped by state
state_totals = data_cleaned.groupby(['State', 'isFraud'])['amount'].sum().unstack(fill_value=0).reset_index()
# Add a new column for the fraud-to-non-fraud ratio
state_totals['fraud_ratio'] = state_totals[1] / (state_totals[0] + 1e-6) # Add small value to avoid division by zero
# Sort by fraud ratio in descending order
state_totals = state_totals.sort_values(by='fraud_ratio', ascending=False)
# Create the bar plot
plt.figure(figsize=(12, 8))
sns.barplot(
y='State',
x='fraud_ratio',
data=state_totals,
palette='viridis'
)
# Annotate the bars with the fraud ratios
for index, row in state_totals.iterrows():
plt.text(
row['fraud_ratio'] + 0.01, index,
f"{row['fraud_ratio']:.2f}",
va='center', ha='left', fontsize=10
)
# Add labels and title
plt.title("Fraud-to-Non-Fraud Transaction Amount Ratio by State", fontsize=16)
plt.xlabel("Fraud-to-Non-Fraud Ratio", fontsize=14)
plt.ylabel("State", fontsize=14)
plt.grid(axis='x', alpha=0.3)
plt.show()
/var/folders/_6/0lw9xh_90gl_6d2dc41byfzh0000gn/T/ipykernel_3138/1047374024.py:12: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(
Insights from the Graph¶
High Fraud-to-Non-Fraud Ratio States:
- Texas, Alaska, and Mississippi have the highest fraud-to-non-fraud transaction ratios.
- These states exhibit a disproportionately high level of fraud relative to non-fraudulent transactions.
Moderate Fraud-to-Non-Fraud Ratio States:
- States like Wyoming, Missouri, and Hawaii show moderate fraud ratios.
- These states maintain a more balanced level of fraud and non-fraud transactions.
Low Fraud-to-Non-Fraud Ratio States:
- Maine, Louisiana, and Connecticut have the lowest fraud-to-non-fraud ratios.
- Fraudulent transactions in these states are minimal when compared to legitimate transactions.
Why This Insight Is Important¶
Fraud Risk Assessment:
- Helps identify states where fraud is significantly more prevalent compared to non-fraud activity.
- Enables organizations to prioritize high-risk regions for targeted interventions.
Geographical Fraud Trends:
- Adjusting for transaction volume, this ratio provides a clearer picture of fraud intensity across states.
Targeted Strategies:
- High-risk states (e.g., Texas) can benefit from stricter fraud detection measures and resource allocation.
In [53]:
# Group the data for fraud transactions by city and calculate the total transaction amount
fraud_city_totals = data_cleaned[data_cleaned['isFraud'] == 1].groupby('merchant_category')['amount'].sum().reset_index()
# Sort by total transaction amount in descending order
fraud_city_totals = fraud_city_totals.sort_values(by='amount', ascending=False)
# Create the bar plot
plt.figure(figsize=(12, 8))
sns.barplot(
y='merchant_category',
x='amount',
data=fraud_city_totals,
palette='viridis'
)
# Add labels and title
plt.title("Total Transaction Amount by Merhants (Fraud Only)", fontsize=16)
plt.xlabel("Total Transaction Amount", fontsize=14)
plt.ylabel("Merchant Category", fontsize=14)
plt.grid(axis='x', alpha=0.3)
plt.show()
/var/folders/_6/0lw9xh_90gl_6d2dc41byfzh0000gn/T/ipykernel_3138/3392577140.py:9: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(
Insights from the Graph¶
Dominant Merchant Categories:
- Groceries is the leading category for fraudulent transactions in terms of total transaction amount. This may indicate a high volume of fraudulent transactions in everyday shopping.
- Electronics follows closely, suggesting that fraudsters target high-value items in this category.
Moderate Fraud Categories:
- Categories such as Entertainment, Apparel, and Travel show significant fraudulent activity, indicating fraud in luxury or discretionary spending.
Low Fraud Categories:
- Automotive and Health categories have the lowest total fraudulent transaction amounts, indicating less frequent targeting by fraudsters.
Why This Insight Is Important¶
Risk Identification:
- The graph highlights which merchant categories are most vulnerable to fraud. High-risk categories like Groceries and Electronics may require stricter monitoring.
Fraud Prevention Strategies:
- Tailored fraud detection measures can be applied to high-risk merchant categories to minimize financial losses.
Understanding Fraud Patterns:
- This categorization can help businesses identify trends and align their fraud detection systems to target specific sectors effectively.
In [55]:
# Calculate the total transaction amount for fraud and non-fraud grouped by state
state_totals = data_cleaned.groupby(['merchant_category', 'isFraud'])['amount'].sum().unstack(fill_value=0).reset_index()
# Add a new column for the fraud-to-non-fraud ratio
state_totals['fraud_ratio'] = state_totals[1] / (state_totals[0] + 1e-6) # Add small value to avoid division by zero
# Sort by fraud ratio in descending order
state_totals = state_totals.sort_values(by='fraud_ratio', ascending=False)
# Create the bar plot
plt.figure(figsize=(12, 8))
sns.barplot(
y='merchant_category',
x='fraud_ratio',
data=state_totals,
palette='viridis'
)
# Annotate the bars with the fraud ratios
for index, row in state_totals.iterrows():
plt.text(
row['fraud_ratio'] + 0.01, index,
f"{row['fraud_ratio']:.2f}",
va='center', ha='left', fontsize=10
)
# Add labels and title
plt.title("Fraud-to-Non-Fraud Transaction Amount Ratio by Merchant", fontsize=16)
plt.xlabel("Fraud-to-Non-Fraud Ratio", fontsize=14)
plt.ylabel("State", fontsize=14)
plt.grid(axis='x', alpha=0.3)
plt.show()
/var/folders/_6/0lw9xh_90gl_6d2dc41byfzh0000gn/T/ipykernel_3138/4152568265.py:12: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(
Insights from the Graph¶
High Fraud-to-Non-Fraud Ratio Categories:
- Automotive and Health categories have the highest fraud-to-non-fraud transaction ratios.
- These categories are particularly vulnerable despite potentially lower overall transaction volumes.
Moderate Fraud-to-Non-Fraud Ratio Categories:
- Categories like Housing, Apparel, and Entertainment show moderate ratios, suggesting fraud is present but not disproportionately higher compared to non-fraud transactions.
Low Fraud-to-Non-Fraud Ratio Categories:
- Groceries and Electronics have relatively lower ratios, indicating a significant number of non-fraud transactions that balance out the fraud transactions.
Risk Prioritization:
- Merchant categories like Automotive and Health require focused fraud prevention efforts due to their disproportionately higher fraud ratios.
Fraud Patterns:
- The graph highlights patterns that help identify categories where fraud occurs at a higher frequency relative to legitimate transactions.
Tailored Fraud Detection:
- Businesses can allocate resources to enhance fraud detection and monitoring systems for high-risk categories while maintaining adequate oversight in moderate-risk ones.
Strategic Decision-Making:
- Understanding these ratios helps businesses and financial institutions design more effective fraud prevention strategies tailored to specific merchant categories.
In [14]:
# Filter data for fraud transactions at night
fraud_night_data = data_cleaned[(data_cleaned['isFraud'] == 1) & (data_cleaned['time_of_day'] == 'Night')]
# Group by 'merchant_category' and calculate total transaction amount
fraud_night_totals = fraud_night_data.groupby('merchant_category')['amount'].sum().reset_index()
# Sort by total transaction amount in descending order
fraud_night_totals = fraud_night_totals.sort_values(by='amount', ascending=False)
# Create the bar plot
plt.figure(figsize=(12, 8))
sns.barplot(
y='merchant_category',
x='amount',
data=fraud_night_totals,
palette='viridis'
)
# Add labels and title
plt.title("Total Fraud Transaction Amount by Merchant Category (Night Only)", fontsize=16)
plt.xlabel("Total Fraud Transaction Amount", fontsize=14)
plt.ylabel("Merchant Category", fontsize=14)
plt.grid(axis='x', alpha=0.3)
plt.show()
/var/folders/_6/0lw9xh_90gl_6d2dc41byfzh0000gn/T/ipykernel_3138/1246375354.py:12: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(
In [15]:
plt.figure(figsize=(10, 8)) #Set figure size for the correlation plot
numerical_columns = data_cleaned.select_dtypes(include=['float64', 'int64']).columns #Select only numerical columns
correlation_matrix = data_cleaned[numerical_columns].corr() #Calculate the correlation matrix for numerical columns
sns.heatmap(
correlation_matrix,
annot=True,
cmap='viridis',
fmt='.2f',
linewidths=0.5
) #Create the heatmap with annotations
plt.title("Correlation Plot for Numerical Columns", fontsize=16) #Add title
plt.show()
isFraudhas weak correlations with most features, indicating that fraudulent behavior might not directly depend on individual numerical variables. Instead, it could rely on a combination of features or categorical data.No Strong Relationships with
isFlaggedFraud:- The column
isFlaggedFraudshows negligible correlations with other features, suggesting that flagged transactions are not strongly influenced by any individual numerical feature.
- The column
Feature Engineering:
- High correlation between
oldbalanceOrgandnewbalanceOrig, as well asoldbalanceDestandnewbalanceDest, could lead to redundancy. It might be beneficial to remove one of these pairs to avoid multicollinearity in machine learning models.
- High correlation between
Fraud Detection Challenges:
- The weak correlations between
isFraudand numerical features indicate that fraudulent transactions might not be easily separable using basic numerical patterns. Advanced methods or additional features may be required to detect fraud effectively.
- The weak correlations between
Focus on Combined Insights:
- Since individual correlations with
isFraudare weak, combining multiple features (e.g., balances and transaction amounts) or adding categorical/contextual data might enhance the model's performance.
- Since individual correlations with
In [16]:
# Separate features and target variable
X = data_cleaned.drop(columns=['isFraud'])
y = data_cleaned['isFraud']
# Define categorical and numerical columns
categorical_columns = ['type', 'State', 'merchant_category', 'time_of_day']
numerical_columns = ['step', 'amount', 'oldbalanceOrg', 'newbalanceOrig', 'oldbalanceDest', 'newbalanceDest', 'isFlaggedFraud']
# Use OneHotEncoder for categorical features
preprocessor = ColumnTransformer(
transformers=[
('num', 'passthrough', numerical_columns),
('cat', OneHotEncoder(handle_unknown='ignore'), categorical_columns)
]
)
# Transform the features
X_transformed = preprocessor.fit_transform(X)
# Apply SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_transformed, y)
# Verify the new class distribution
print("\nClass distribution after SMOTE:")
print(pd.Series(y_resampled).value_counts(normalize=True))
# Combine resampled features and target variable into a DataFrame
# Convert OneHotEncoder's sparse matrix to dense DataFrame
X_resampled_df = pd.DataFrame.sparse.from_spmatrix(X_resampled, columns=preprocessor.get_feature_names_out())
X_resampled_df['isFraud'] = y_resampled
Class distribution after SMOTE: isFraud 0 0.5 1 0.5 Name: proportion, dtype: float64
Data cleaning and Preprocessing¶
Feature and Target Separation:
- Splits the data into features (X) and the target variable (y), where y represents whether a transaction is fraudulent (isFraud).
Preprocessing:
- Defines numerical and categorical columns.
- Uses OneHotEncoder to encode categorical features into a numerical format while passing numerical features as they are.
- One-Hot Encoding is a technique used to convert categorical data into a numerical format by creating binary columns for each unique category. Each category is represented by a column, where a value of 1 indicates the presence of the category, and 0 indicates its absence.
Handling Imbalance
As we see, the data replicates real-life scenarios of the highly imbalanced datasets for frauds (less than 1% being a fraud), but we cannot use this data directly due to such a high imbalance
Hence, we used a technique called SMOTE ((Synthetic Minority Oversampling Technique), which will add synthetic data points for the minority class that is in this class isFraud=1
SMOTE (Synthetic Minority Oversampling Technique) is a technique used to address class imbalance by generating synthetic samples for the minority class. It creates new data points by interpolating between existing samples, ensuring a balanced dataset for better model performance.
RESULT¶
- We now have a balanced dataset with new values of distribution as follows
- Class distribution after SMOTE: Fraud=0.5 , Not Fraud=0.5 , showing an equal split
In [15]:
X_resampled_df
Out[15]:
| num__step | num__amount | num__oldbalanceOrg | num__newbalanceOrig | num__oldbalanceDest | num__newbalanceDest | num__isFlaggedFraud | cat__type_CASH_IN | cat__type_CASH_OUT | cat__type_DEBIT | ... | cat__merchant_category_Entertainment | cat__merchant_category_Groceries | cat__merchant_category_Health | cat__merchant_category_Housing | cat__merchant_category_Travel | cat__time_of_day_Afternoon | cat__time_of_day_Evening | cat__time_of_day_Morning | cat__time_of_day_Night | isFraud | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1.0 | 1864.28 | 14267.73 | 19384.72 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 1.0 | 0 | 0 | 0 | 0 | 1.0 | 0 |
| 1 | 1.0 | 181.0 | 181.0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 1.0 | 0 | 0 | 0 | 0 | 1.0 | 1 |
| 2 | 1.0 | 181.0 | 181.0 | 0 | 21182.0 | 0 | 0 | 0 | 1.0 | 0 | ... | 0 | 1.0 | 0 | 0 | 0 | 0 | 0 | 0 | 1.0 | 1 |
| 3 | 1.0 | 7817.71 | 53860.0 | 46042.29 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 1.0 | 0 | 0 | 0 | 0 | 0 | 1.0 | 0 | 0 | 0 |
| 4 | 1.0 | 7107.77 | 183195.0 | 176087.23 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 1.0 | 0 | 0 | 0 | 0 | 1.0 | 0 | 0 | 0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 9259241 | 504.681465 | 434357.349406 | 434357.349406 | 0 | 4184524.834593 | 4618882.177981 | 0 | 0 | 1.0 | 0 | ... | 0 | 0.398229 | 0 | 0 | 0.601771 | 0 | 0 | 0.398229 | 0.601771 | 1 |
| 9259242 | 425.386743 | 4686519.883199 | 4686519.883199 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0.316906 | 0 | 0 | 0 | 0 | 0 | 0 | 1.0 | 1 |
| 9259243 | 390.127487 | 249085.739251 | 249085.739251 | 0 | 0 | 249085.739251 | 0 | 0 | 1.0 | 0 | ... | 1.0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1.0 | 1 |
| 9259244 | 307.91172 | 2374456.908931 | 2374456.908931 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0.432177 | 0 | 0 | 0 | 1.0 | 1 |
| 9259245 | 559.341213 | 55559.610731 | 55559.610731 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0.068243 | 0 | 0 | 0.931757 | 0 | 0 | 0 | 1.0 | 1 |
9259246 rows × 75 columns
In [70]:
#Predictive Modeling
In [2]:
file_path = "final_cleaned_dataset_with_smote.csv"
data = pd.read_csv(file_path)
# Separate features (X) and target variable (y)
X = data.drop(columns=['isFraud'])
y = data['isFraud']
# Step 2: Scale features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Step 3: Split data
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42, stratify=y)
# Step 4: Train a Logistic Regression model with increased iterations and adjusted solver
model = LogisticRegression(masmote
x_iter=1000, solver='saga', random_state=42)
model.fit(X_train, y_train)
# Step 5: Make predictions
y_pred = model.predict(X_test)
y_pred_proba = model.predict_proba(X_test)[:, 1]
# Step 6: Evaluate the model
print("\nClassification Report:")
print(classification_report(y_test, y_pred))
print("\nROC-AUC Score:")
print(roc_auc_score(y_test, y_pred_proba))
/opt/anaconda3/lib/python3.12/site-packages/sklearn/linear_model/_sag.py:350: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge warnings.warn(
Classification Report:
precision recall f1-score support
0 0.97 0.97 0.97 925925
1 0.97 0.97 0.97 925925
accuracy 0.97 1851850
macro avg 0.97 0.97 0.97 1851850
weighted avg 0.97 0.97 0.97 1851850
ROC-AUC Score:
0.9956106770250465
Models, algorithms and training¶
Linear Model - Logistic Regression(Solver=SAGA)
After necessary preprocessing, we start with training the model with Logistic Regression, with solver='saga'
SAGA is an optimization algorithm used in logistic regression and other linear models designed for large-scale datasets. It is particularly efficient for datasets with sparse features (many zeros) and supports L1 (Lasso), L2 (Ridge), and Elastic Net regularization. SAGA improves over older algorithms like SGD by achieving faster convergence and better stability.
we get a High accuracy on training and test data which clarifies that the data is not overfitting for this algorithm ROC-AUC score=0.99 and accuracy=0.97
In [4]:
# Import necessary libraries
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
# Train Logistic Regression with higher iterations and alternative solver
model = LogisticRegression(max_iter=2000, solver='lbfgs', random_state=42)
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
y_pred_proba = model.predict_proba(X_test)[:, 1]
# Evaluate the model
print("\nClassification Report:")
print(classification_report(y_test, y_pred))
print("\nROC-AUC Score:")
print(roc_auc_score(y_test, y_pred_proba))
Classification Report:
precision recall f1-score support
0 0.98 0.97 0.97 925925
1 0.97 0.98 0.97 925925
accuracy 0.97 1851850
macro avg 0.97 0.97 0.97 1851850
weighted avg 0.97 0.97 0.97 1851850
ROC-AUC Score:
0.9960639279265301
Models, algorithms and training¶
Linear Model - Logistic Regression(solver=lbfgs)
After necessary preprocessing, we start with training the model with Logistic Regression, with solver='lbfgs'
L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) is an optimization algorithm used in logistic regression and other models for parameter estimation. It is efficient for solving problems with dense data and moderate-sized datasets. L-BFGS approximates the Hessian matrix (second-order derivatives) without storing it fully, making it computationally faster and memory-efficient compared to full Newton methods.
we get a High accuracy on training and test data which clarifies that the data is not overfitting for this algorithm ROC-AUC score=0.99 and accuracy=0.97
we do not see much difference between the two solvers except for execution time for L-BFGS being faster
In [6]:
#train and evaluate both Random Forest and XGBoost models
In [14]:
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from sklearn.metrics import classification_report, roc_auc_score
# Random Forest Model
print("Training Random Forest...")
rf_model = RandomForestClassifier(
n_estimators=100, # Number of trees in the forest
max_depth=None, # Allow trees to expand fully
random_state=42, # Ensure reproducibility
n_jobs=-1 # Use all available cores for parallel processing
)
rf_model.fit(X_train, y_train)
# Random Forest Predictions
rf_y_pred = rf_model.predict(X_test)
rf_y_pred_proba = rf_model.predict_proba(X_test)[:, 1]
# Evaluate Random Forest
print("\nRandom Forest Classification Report:")
print(classification_report(y_test, rf_y_pred))
print("\nRandom Forest ROC-AUC Score:")
print(roc_auc_score(y_test, rf_y_pred_proba))
Training Random Forest...
Random Forest Classification Report:
precision recall f1-score support
0 1.00 1.00 1.00 925925
1 1.00 1.00 1.00 925925
accuracy 1.00 1851850
macro avg 1.00 1.00 1.00 1851850
weighted avg 1.00 1.00 1.00 1851850
Random Forest ROC-AUC Score:
0.9999981903967647
Models, algorithms and training¶
Non Linear Model - Random forest
After trying with Linear Models, we can now try with Non-linear models with effective parameters to see the performance here, we have trained a Random Forest Classifier
Exceptional Performance:
- The Random Forest model achieves near-perfect performance with an accuracy of 100% and an ROC-AUC score of 0.9999.
- Both precision and recall for the fraud class (
isFraud = 1) are extremely high, indicating the model's ability to correctly identify fraudulent transactions without misclassifying non-fraudulent ones.
Balanced Results:
The classification report shows that the model performs equally well for both classes (
isFraud = 0andisFraud = 1), with high F1-scores and balanced metrics.with an accuracy of 0.99, we concluded Non-Linear models have performed marginally better than Linear Models
In [15]:
# XGBoost Model (Updated)
print("\nTraining XGBoost...")
xgb_model = XGBClassifier(
n_estimators=100, # Number of boosting rounds
random_state=42, # Ensure reproducibility
eval_metric='logloss', # Evaluation metric for classification
tree_method='hist', # Efficient histogram-based algorithm
n_jobs=-1 # Use all available cores for parallel processing
)
xgb_model.fit(X_train, y_train)
# XGBoost Predictions
xgb_y_pred = xgb_model.predict(X_test)
xgb_y_pred_proba = xgb_model.predict_proba(X_test)[:, 1]
# Evaluate XGBoost
print("\nXGBoost Classification Report:")
print(classification_report(y_test, xgb_y_pred))
print("\nXGBoost ROC-AUC Score:")
print(roc_auc_score(y_test, xgb_y_pred_proba))
Training XGBoost...
XGBoost Classification Report:
precision recall f1-score support
0 1.00 1.00 1.00 925925
1 1.00 1.00 1.00 925925
accuracy 1.00 1851850
macro avg 1.00 1.00 1.00 1851850
weighted avg 1.00 1.00 1.00 1851850
XGBoost ROC-AUC Score:
0.9999978503257494
Models, algorithms and training¶
Non Linear Model - XGBOOST
After trying with Linear Models, we can now try with Non-linear models with effective parameters to see the performance here, we have trained an XGBOOST
Exceptional Performance:
- The XGBOOST model achieves near-perfect performance with an accuracy of 100% and an ROC-AUC score of 0.9999.
- Both precision and recall for the fraud class (
isFraud = 1) are extremely high, indicating the model's ability to correctly identify fraudulent transactions without misclassifying non-fraudulent ones.
Balanced Results:
- The classification report shows that the model performs equally well for both classes (
isFraud = 0andisFraud = 1), with high F1-scores and balanced metrics. - with an accuracy of 0.99, we concluded Non-linear models have performed marginally better than Linear Models
- The classification report shows that the model performs equally well for both classes (
Important ******¶
Exceptionally high accuracy for Non-Linear Models can be explained due to the following reasons
- Huge Dataset- total rows about 9300000, which is good enough for any model to train thoroughly
- Due to its highly imbalanced nature, we applied SMOTE, making the splits equal, that is, 0.5 for fraud and 0.5 for not fraud, which results in this exceptionally high accuracy
- Both the Training and Test errors are very low, indicating a low bias and low variance and hence nullifying the possibility of Overfitting for any of the algorithms
- Using Stratified sampling and ensuring the datasets are different has also nullified the possibility of Data Leakage
- Hence ,we conclude this exceptionally high accuracy valid due to the above reasons for Non-Linear Models
In [18]:
#Feature Importance Analysis:
In [22]:
# Feature Importance for Random Forest
print("\nRandom Forest Feature Importance:")
rf_feature_importance = pd.DataFrame({
'Feature': X.columns,
'Importance': rf_model.feature_importances_
}).sort_values(by='Importance', ascending=False)
# Feature Importance for XGBoost
print("\nXGBoost Feature Importance:")
xgb_feature_importance = pd.DataFrame({
'Feature': X.columns,
'Importance': xgb_model.feature_importances_
}).sort_values(by='Importance', ascending=False)
In [24]:
# Select important features (Importance > 1% in either model)
important_features = list(set(
rf_feature_importance[rf_feature_importance['Importance'] > 0.01]['Feature'].tolist() +
xgb_feature_importance[xgb_feature_importance['Importance'] > 0.01]['Feature'].tolist()
))
print("\nSelected Features for the Simplified Model:")
print(important_features)
# Subset the data to keep only the selected features
X_refined = X[important_features]
Selected Features for the Simplified Model: ['cat__type_PAYMENT', 'cat__time_of_day_Evening', 'num__amount', 'num__oldbalanceOrg', 'num__newbalanceOrig', 'cat__State_Texas', 'cat__type_CASH_IN', 'num__step', 'num__isFlaggedFraud', 'num__newbalanceDest', 'num__oldbalanceDest', 'cat__time_of_day_Night', 'cat__time_of_day_Afternoon', 'cat__type_CASH_OUT', 'cat__type_TRANSFER']
In [26]:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from sklearn.metrics import classification_report, roc_auc_score
# Load Dataset
file_path = "Fraud Dataset.csv" # Replace with actual path
data = pd.read_csv(file_path)
# Drop rows with missing values to ensure consistency
data = data.dropna()
# Define Target Variable (y) and Features (X)
important_features_raw = [
'type', 'time_of_day', 'amount',
'oldbalanceOrg', 'newbalanceOrig', 'State',
'isFlaggedFraud', 'step', 'newbalanceDest',
'oldbalanceDest'
]
X_refined = pd.get_dummies(data[important_features_raw], drop_first=True) # One-hot encoding for categorical features
y = data['isFraud']
# Verify Sizes
print("Shape of X_refined:", X_refined.shape)
print("Length of y:", len(y))
# Split Dataset
X_train_refined, X_test_refined, y_train, y_test = train_test_split(
X_refined, y, test_size=0.2, random_state=42, stratify=y
)
# Train and Evaluate Random Forest
print("\nTraining Random Forest with Simplified Features...")
rf_model_refined = RandomForestClassifier(
n_estimators=100, random_state=42, n_jobs=-1
)
rf_model_refined.fit(X_train_refined, y_train)
# Predictions
rf_y_pred_refined = rf_model_refined.predict(X_test_refined)
rf_y_pred_proba_refined = rf_model_refined.predict_proba(X_test_refined)[:, 1]
# Evaluate
print("\nRandom Forest (Simplified) Classification Report:")
print(classification_report(y_test, rf_y_pred_refined))
print("\nRandom Forest (Simplified) ROC-AUC Score:")
print(roc_auc_score(y_test, rf_y_pred_proba_refined))
# Train and Evaluate XGBoost
print("\nTraining XGBoost with Simplified Features...")
xgb_model_refined = XGBClassifier(
n_estimators=100, random_state=42, eval_metric='logloss', tree_method='hist', n_jobs=-1
)
xgb_model_refined.fit(X_train_refined, y_train)
# Predictions
xgb_y_pred_refined = xgb_model_refined.predict(X_test_refined)
xgb_y_pred_proba_refined = xgb_model_refined.predict_proba(X_test_refined)[:, 1]
# Evaluate
print("\nXGBoost (Simplified) Classification Report:")
print(classification_report(y_test, xgb_y_pred_refined))
print("\nXGBoost (Simplified) ROC-AUC Score:")
print(roc_auc_score(y_test, xgb_y_pred_proba_refined))
Shape of X_refined: (1315551, 63)
Length of y: 1315551
Training Random Forest with Simplified Features...
Random Forest (Simplified) Classification Report:
precision recall f1-score support
0 1.00 1.00 1.00 261468
1 0.98 0.81 0.89 1643
accuracy 1.00 263111
macro avg 0.99 0.90 0.94 263111
weighted avg 1.00 1.00 1.00 263111
Random Forest (Simplified) ROC-AUC Score:
0.9972647158050392
Training XGBoost with Simplified Features...
XGBoost (Simplified) Classification Report:
precision recall f1-score support
0 1.00 1.00 1.00 261468
1 0.96 0.93 0.95 1643
accuracy 1.00 263111
macro avg 0.98 0.97 0.97 263111
weighted avg 1.00 1.00 1.00 263111
XGBoost (Simplified) ROC-AUC Score:
0.9996894122246116
Models, algorithms and training¶
Let us try to train the data with the same Algorithms but this time using Feature importance and feature selection and observe if we see any difference in the accuracy, precision, recall, f1-score, or accuracy
The feature importance in this code is calculated using Gini impurity for Random Forest and Gain for XGBoost by default
Upon training the model with Random Forest with feature selection and limited features f-1 score for fraud earlier- 1.00, after feature importance f-1 score=0.89, we see a dip in the f-1 score, which states that the features we had assumed earlier are all required
Upon training the model with Random Forest with feature selection and limited features f-1 score for fraud earlier- 1.00, after feature importance f-1 score=0.95, we see a dip in the f-1 score, which states that the features we had assumed earlier are all required
Feature importance scores don’t always capture synergistic relationships between features, so removing features based purely on individual importance might hurt the model.
In [28]:
#K-Fold Cross-Validation
In [32]:
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
# Define important features and target variable
important_features_raw = [
'type', 'time_of_day', 'amount',
'oldbalanceOrg', 'newbalanceOrig', 'State',
'isFlaggedFraud', 'step', 'newbalanceDest',
'oldbalanceDest'
]
X = pd.get_dummies(data[important_features_raw], drop_first=True)
y = data['isFraud']
# Random Forest with Cross-Validation
print("\nEvaluating Random Forest with Cross-Validation...")
rf_model = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
rf_cv_scores = cross_val_score(rf_model, X, y, cv=5, scoring='roc_auc') # 5-Fold CV
print("Random Forest Cross-Validation AUC Scores:", rf_cv_scores)
print("Random Forest Average AUC Score:", rf_cv_scores.mean())
# XGBoost with Cross-Validation
print("\nEvaluating XGBoost with Cross-Validation...")
xgb_model = XGBClassifier(n_estimators=100, random_state=42, eval_metric='logloss', tree_method='hist', n_jobs=-1)
xgb_cv_scores = cross_val_score(xgb_model, X, y, cv=5, scoring='roc_auc') # 5-Fold CV
print("XGBoost Cross-Validation AUC Scores:", xgb_cv_scores)
print("XGBoost Average AUC Score:", xgb_cv_scores.mean())
Evaluating Random Forest with Cross-Validation... Random Forest Cross-Validation AUC Scores: [0.98595782 0.98361123 0.94292288 0.98732157 0.99691797] Random Forest Average AUC Score: 0.9793462951320222 Evaluating XGBoost with Cross-Validation... XGBoost Cross-Validation AUC Scores: [0.99630813 0.99734344 0.99242596 0.99870832 0.99983086] XGBoost Average AUC Score: 0.9969233407803209
Models, algorithms and training¶
Cross-Validation Results and Analysis¶
1. Random Forest with Cross-Validation¶
AUC Scores:
- Random Forest achieves AUC scores ranging from 0.94 to 0.99 across the 5-fold cross-validation splits.
- The variability in scores (e.g., 0.94 in one fold vs. 0.99 in others) might indicate slight sensitivity to the specific data split for that fold and not affect the entire performance.
Average AUC:
- The average AUC score of 0.9793 demonstrates strong performance, indicating that Random Forest effectively separates fraudulent and non-fraudulent transactions.
2. XGBoost with Cross-Validation¶
AUC Scores:
- XGBoost shows consistently high AUC scores, ranging from 0.99 to nearly 1.00, across all folds.
- This suggests that XGBoost is highly robust and performs well on each validation split, likely due to its advanced regularization techniques and optimization.
Average AUC:
- With an average AUC score of 0.9969, XGBoost outperforms Random Forest in this scenario, demonstrating superior discrimination between fraud and non-fraud cases.
In [38]:
#Hyperparameter Tuning
In [40]:
from sklearn.model_selection import GridSearchCV
# Define parameter grid for XGBoost
xgb_param_grid = {
'n_estimators': [100, 200],
'max_depth': [3, 5, 7],
'learning_rate': [0.01, 0.1, 0.2],
'subsample': [0.8, 1.0],
}
# XGBoost with GridSearchCV
print("\nHyperparameter Tuning for XGBoost...")
xgb_model = XGBClassifier(random_state=42, tree_method='hist', eval_metric='logloss')
xgb_grid = GridSearchCV(estimator=xgb_model, param_grid=xgb_param_grid, cv=3, scoring='roc_auc', n_jobs=-1)
xgb_grid.fit(X_train_refined, y_train)
print("\nBest Parameters for XGBoost:", xgb_grid.best_params_)
print("Best ROC-AUC Score:", xgb_grid.best_score_)
# LightGBM Hyperparameter Tuning
from lightgbm import LGBMClassifier
lgbm_param_grid = {
'n_estimators': [100, 200],
'max_depth': [-1, 5, 10],
'learning_rate': [0.01, 0.1],
'num_leaves': [31, 50],
}
print("\nHyperparameter Tuning for LightGBM...")
lgbm_model = LGBMClassifier(random_state=42)
lgbm_grid = GridSearchCV(estimator=lgbm_model, param_grid=lgbm_param_grid, cv=3, scoring='roc_auc', n_jobs=-1)
lgbm_grid.fit(X_train_refined, y_train)
print("\nBest Parameters for LightGBM:", lgbm_grid.best_params_)
print("Best ROC-AUC Score:", lgbm_grid.best_score_)
Hyperparameter Tuning for XGBoost...
Best Parameters for XGBoost: {'learning_rate': 0.1, 'max_depth': 5, 'n_estimators': 200, 'subsample': 1.0}
Best ROC-AUC Score: 0.9996020249950327
Hyperparameter Tuning for LightGBM...
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.010146 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.016171 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.015770 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.048094 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.005846 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.005883 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.020454 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.017815 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.004750 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.012804 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.004705 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.004413 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.004020 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.060410 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.008173 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.016719 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.013306 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.013208 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.008397 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.003745 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.011785 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.006321 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.013043 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.005856 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.010480 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.004747 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.045016 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.006489 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.013377 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697247
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.012908 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701627, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 4380, number of negative: 697246
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.003744 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1642
[LightGBM] [Info] Number of data points in the train set: 701626, number of used features: 62
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070090
[LightGBM] [Info] Start training from score -5.070090
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Number of positive: 6570, number of negative: 1045870
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.003633 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1644
[LightGBM] [Info] Number of data points in the train set: 1052440, number of used features: 63
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.006243 -> initscore=-5.070091
[LightGBM] [Info] Start training from score -5.070091
Best Parameters for LightGBM: {'learning_rate': 0.01, 'max_depth': 10, 'n_estimators': 200, 'num_leaves': 50}
Best ROC-AUC Score: 0.9992145229600798
Models, algorithms and training¶
Hypter-Parameter Tuning¶
- Let us Experiment with some of the probable best hyperparameters for our best Model XGBOOST to see if we can further improve the performance by a slight margin
Best Parameters for XGBoost: {'learning_rate': 0.1, 'max_depth': 5, 'n_estimators': 200, 'subsample': 1.0} Best ROC-AUC Score: 0.999
LightGBM (Light Gradient Boosting Machine):
- A highly efficient implementation of gradient boosting developed by Microsoft.
- It is designed for speed and performance, particularly for large datasets and high-dimensional feature spaces.
Best Parameters for LightGBM: {'learning_rate': 0.01, 'max_depth': 10, 'n_estimators': 200, 'num_leaves': 50} Best ROC-AUC Score: 0.9992145229600798
Hence, comparing our new LightGBM classifier and XGBoost, XGBoost remains our best model, performing marginally better than LighGBM
In [42]:
#Deployment Preparation
import joblib
# Save the best XGBoost model
print("\nSaving XGBoost Model...")
joblib.dump(xgb_grid.best_estimator_, "final_xgb_model.pkl")
# Save the best LightGBM model
print("\nSaving LightGBM Model...")
joblib.dump(lgbm_grid.best_estimator_, "final_lgbm_model.pkl")
# Load the saved model for inference (example)
loaded_model = joblib.load("final_xgb_model.pkl")
# Example Inference
sample_data = X_test_refined.iloc[:5] # Replace with actual unseen data
predictions = loaded_model.predict(sample_data)
probabilities = loaded_model.predict_proba(sample_data)[:, 1]
print("\nPredictions on Sample Data:", predictions)
print("Probabilities on Sample Data:", probabilities)
Saving XGBoost Model... Saving LightGBM Model... Predictions on Sample Data: [0 0 0 0 0] Probabilities on Sample Data: [1.2146346e-06 4.0518964e-07 1.4092450e-06 6.3313141e-06 9.2189019e-08]
Validation of Sample Data: Ensure the predictions align with the actual labels of the sample data to validate model accuracy.
Model Selection and Finalization¶
After extensive experimentation and evaluation of multiple models using the following methods:
- Cross-Validation: Trained models with multiple folds to ensure generalizability.
- Grid Search CV: Tuned hyperparameters to optimize performance.
- Feature Importance: Assessed the contribution of individual features to enhance model interpretability.
We carefully analyzed key performance metrics, including:
- F1-Score
- ROC-AUC Score
- Accuracy
- Precision
- Recall
Among all tested models, XGBoost consistently outperformed others, demonstrating superior performance across these metrics. Even after rigorous training and evaluation, including feature refinement and hyperparameter tuning, XGBoost emerged as the most robust and reliable model for our fraud detection task.
As a result, XGBoost has been selected as the final model. (ROC-AUC Curve comparison between the two graphs can be seen visually below) The model was saved using both joblib and pickle for future deployment and inference.
In [45]:
# Check actual labels for the sample data
print("Actual Labels for Sample Data:")
print(y_test.iloc[:5].values)
Actual Labels for Sample Data: [0 0 0 0 0]
Inference Pipeline: Prepare an end-to-end pipeline for inference. The pipeline will take raw input data, process it to match the model's requirements, and generate predictions.
In [51]:
def fraud_inference_pipeline(model_path, input_data, feature_columns):
"""
Fraud detection inference pipeline.
:param model_path: Path to the saved model.
:param input_data: Raw input data (DataFrame).
:param feature_columns: List of feature columns used during training.
:return: Predictions and probabilities.
"""
# Load the model
model = joblib.load(model_path)
# Preprocess the input data to match the training data
processed_data = pd.get_dummies(input_data, drop_first=True)
processed_data = processed_data.reindex(columns=feature_columns, fill_value=0)
# Generate predictions
predictions = model.predict(processed_data)
probabilities = model.predict_proba(processed_data)[:, 1]
return predictions, probabilities
# Example usage
# Ensure that `X_test_refined` and `X_test` are loaded/available for testing
sample_data = X_test_refined.iloc[:5] # This assumes you have split your data into train and test sets
feature_columns = X_train_refined.columns # Pass the feature columns used during training
# Call the pipeline
predictions, probabilities = fraud_inference_pipeline("final_xgb_model.pkl", sample_data, feature_columns)
# Output the predictions
print("\nPipeline Predictions:", predictions)
print("Pipeline Probabilities:", probabilities)
Pipeline Predictions: [0 0 0 0 0] Pipeline Probabilities: [1.2146346e-06 4.0518964e-07 1.4092450e-06 6.3313141e-06 9.2189019e-08]
Understanding the Output¶
1. Predictions on Sample Data:¶
[0, 0, 0, 0, 0]
- The predict() method outputs the predicted class for each sample:
- 0 indicates that the model predicts the negative class (e.g., isFraud = 0).
- All predictions are 0, suggesting that the model does not detect fraud in these samples.
2. Probabilities on Sample Data:¶
[1.2146346e-06, 4.0518964e-07, 1.4092450e-06, 6.3313141e-06, 9.2189019e-08]
- The predict_proba() method outputs the probabilities for each class:
- [:, 1] gives the probability of the sample belonging to the positive class (e.g., isFraud = 1).
- The probabilities are very close to 0, confirming that the model has low confidence in these samples being fraudulent.
Why Are Probabilities Extremely Small?¶
1. Nature of the Data¶
- Fraud detection datasets are often highly imbalanced, with very few fraudulent transactions.
- The model may learn to assign very low probabilities to the positive class unless strong fraud indicators are present.
2. Model Behavior¶
- The model assigns probabilities based on learned patterns.
- If the input data does not exhibit fraud-like characteristics, the probabilities for isFraud = 1 will be near zero.
In [57]:
#Visualization and Markdown Explanation
In [59]:
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
# 1. Feature Importance Visualization for XGBoost
def plot_feature_importance(feature_importance, title):
plt.figure(figsize=(10, 6))
feature_importance.sort_values(by="Importance", ascending=False, inplace=True)
sns.barplot(x="Importance", y="Feature", data=feature_importance)
plt.title(title)
plt.xlabel("Importance Score")
plt.ylabel("Features")
plt.show()
# Create feature importance DataFrame for XGBoost
xgb_feature_importance = pd.DataFrame({
"Feature": X_train_refined.columns,
"Importance": xgb_model_refined.feature_importances_
})
plot_feature_importance(xgb_feature_importance, "XGBoost Feature Importance")
# 2. Fraud Distribution by Transaction Type
def plot_fraud_distribution_by_type(data):
fraud_rates = data.groupby("type")["isFraud"].mean().sort_values()
fraud_rates.plot(kind="bar", figsize=(8, 5), color="skyblue", edgecolor="black")
plt.title("Fraud Rates by Transaction Type")
plt.ylabel("Fraud Rate")
plt.xlabel("Transaction Type")
plt.show()
plot_fraud_distribution_by_type(data)
# 3. Confusion Matrix for XGBoost
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
def plot_confusion_matrix(y_true, y_pred, model_name):
cm = confusion_matrix(y_true, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["Non-Fraud", "Fraud"])
disp.plot(cmap="Blues")
plt.title(f"Confusion Matrix: {model_name}")
plt.show()
plot_confusion_matrix(y_test, xgb_y_pred_refined, "XGBoost")
# 4. ROC Curve for XGBoost
from sklearn.metrics import roc_curve, auc
def plot_roc_curve(y_true, y_proba, model_name):
fpr, tpr, thresholds = roc_curve(y_true, y_proba)
roc_auc = auc(fpr, tpr)
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f"AUC = {roc_auc:.2f}")
plt.plot([0, 1], [0, 1], "r--")
plt.title(f"ROC Curve: {model_name}")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.legend(loc="lower right")
plt.show()
plot_roc_curve(y_test, xgb_y_pred_proba_refined, "XGBoost")
Feature Importance: The bar plot highlights the relative importance of each feature in the XGBoost model. Features such as num__amount, num__oldbalanceOrg, and num__newbalanceOrig stand out as key drivers for fraud detection.
Fraud Distribution by Transaction Type: A bar chart showcasing fraud rates by transaction type provides actionable insights. For instance, TRANSFER and CASH_OUT exhibit significantly higher fraud rates compared to others.
Confusion Matrix: The confusion matrix visualizes the model's classification performance, including true positives, true negatives, false positives, and false negatives.
ROC Curve: The ROC curve and AUC score evaluate the model's ability to distinguish between fraudulent and non-fraudulent transactions. A near-perfect AUC score underscores the model's robustness.
In [63]:
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay, roc_curve, auc
# Example data for visualization (replace with actual data)
xgb_feature_importance = pd.DataFrame({
"Feature": X_train_refined.columns,
"Importance": xgb_model_refined.feature_importances_
})
# Plot 1: Feature Importance for XGBoost
def plot_feature_importance(feature_importance, title):
plt.figure(figsize=(10, 6))
feature_importance.sort_values(by="Importance", ascending=False, inplace=True)
sns.barplot(x="Importance", y="Feature", data=feature_importance, palette="viridis")
plt.title(title)
plt.xlabel("Importance Score")
plt.ylabel("Features")
plt.tight_layout()
plt.show()
plot_feature_importance(xgb_feature_importance, "XGBoost Feature Importance")
# Plot 2: Fraud Distribution by Transaction Type
def plot_fraud_distribution_by_type(data):
fraud_rates = data.groupby("type")["isFraud"].mean().sort_values()
fraud_rates.plot(kind="bar", figsize=(8, 5), color="skyblue", edgecolor="black")
plt.title("Fraud Rates by Transaction Type")
plt.ylabel("Fraud Rate")
plt.xlabel("Transaction Type")
plt.tight_layout()
plt.show()
# Replace 'data' with the actual dataset
plot_fraud_distribution_by_type(data)
# Plot 3: Confusion Matrix for XGBoost
def plot_confusion_matrix(y_true, y_pred, model_name):
cm = confusion_matrix(y_true, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["Non-Fraud", "Fraud"])
disp.plot(cmap="Blues")
plt.title(f"Confusion Matrix: {model_name}")
plt.tight_layout()
plt.show()
plot_confusion_matrix(y_test, xgb_y_pred_refined, "XGBoost")
# Plot 4: ROC Curve for XGBoost
def plot_roc_curve(y_true, y_proba, model_name):
fpr, tpr, thresholds = roc_curve(y_true, y_proba)
roc_auc = auc(fpr, tpr)
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f"AUC = {roc_auc:.2f}", color='blue')
plt.plot([0, 1], [0, 1], "r--")
plt.title(f"ROC Curve: {model_name}")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.legend(loc="lower right")
plt.tight_layout()
plt.show()
plot_roc_curve(y_test, xgb_y_pred_proba_refined, "XGBoost")
/var/folders/sq/bfcsfb3x1ndczwvbpfn7f41c0000gn/T/ipykernel_55859/473583233.py:17: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x="Importance", y="Feature", data=feature_importance, palette="viridis")
Analysis of Confusion Matrix (XGBoost)¶
1. Interpretation of Values¶
- True Negatives (261,409):
- The model correctly classified 261,409 non-fraudulent transactions as non-fraudulent.
- False Positives (59):
- The model incorrectly classified 59 non-fraudulent transactions as fraudulent.
- True Positives (1,536):
- The model correctly identified 1,536 fraudulent transactions as fraud.
- False Negatives (107):
- The model missed 107 fraudulent transactions, predicting them as non-fraudulent.
2. Key Metrics Derived¶
- High True Negative Count:
- The model is very effective at identifying legitimate transactions, with a low rate of false positives.
- High True Positive Count:
- It performs well at detecting fraudulent transactions, correctly flagging the majority of them.
- Low False Positives and Negatives:
- A small number of non-fraudulent transactions are misclassified as fraud, and relatively few frauds are missed.
In [65]:
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import ConfusionMatrixDisplay, roc_curve, auc, confusion_matrix
import numpy as np
# Enhanced Feature Importance Plot
def improved_feature_importance_plot(feature_importance, title, top_n=10):
feature_importance = feature_importance.sort_values(by="Importance", ascending=True).tail(top_n)
plt.figure(figsize=(10, 8))
sns.barplot(x="Importance", y="Feature", data=feature_importance, palette="viridis")
plt.title(title, fontsize=16)
plt.xlabel("Importance Score", fontsize=14)
plt.ylabel("Features", fontsize=14)
for index, row in feature_importance.iterrows():
plt.text(row['Importance'], index, f"{row['Importance']:.2f}", va='center', fontsize=10)
plt.tight_layout()
plt.show()
# Improved Fraud Distribution Plot
def improved_fraud_distribution_plot(data):
fraud_rates = data.groupby("type")["isFraud"].mean().sort_values()
fraud_counts = data["type"].value_counts()
plt.figure(figsize=(12, 6))
sns.barplot(x=fraud_rates.index, y=fraud_rates.values, palette="magma")
plt.title("Fraud Rates by Transaction Type", fontsize=16)
plt.ylabel("Fraud Rate", fontsize=14)
plt.xlabel("Transaction Type", fontsize=14)
for i, (rate, count) in enumerate(zip(fraud_rates.values, fraud_counts[fraud_rates.index])):
plt.text(i, rate + 0.005, f"{rate:.2%}\n(n={count})", ha="center", fontsize=10)
plt.xticks(rotation=45, fontsize=12)
plt.tight_layout()
plt.show()
# Improved Confusion Matrix Plot
def improved_confusion_matrix_plot(y_true, y_pred, model_name):
cm = confusion_matrix(y_true, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["Non-Fraud", "Fraud"])
disp.plot(cmap="Blues", values_format="d")
plt.title(f"Confusion Matrix: {model_name}", fontsize=16)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)
plt.tight_layout()
plt.show()
# Improved ROC Curve for Multiple Models
def improved_roc_curve_plot(models, y_true, y_proba_list, model_names):
plt.figure(figsize=(12, 8))
for y_proba, name in zip(y_proba_list, model_names):
fpr, tpr, _ = roc_curve(y_true, y_proba)
roc_auc = auc(fpr, tpr)
plt.plot(fpr, tpr, label=f"{name} AUC = {roc_auc:.3f}")
plt.plot([0, 1], [0, 1], "r--", lw=2)
plt.title("ROC Curve Comparison", fontsize=16)
plt.xlabel("False Positive Rate", fontsize=14)
plt.ylabel("True Positive Rate", fontsize=14)
plt.legend(loc="lower right", fontsize=12)
plt.grid(True)
plt.tight_layout()
plt.show()
# Call the improved visualization functions
improved_feature_importance_plot(xgb_feature_importance, "XGBoost Feature Importance", top_n=10)
improved_fraud_distribution_plot(data)
improved_confusion_matrix_plot(y_test, xgb_y_pred_refined, "XGBoost")
improved_roc_curve_plot(
models=["XGBoost", "Random Forest"],
y_true=y_test,
y_proba_list=[xgb_y_pred_proba_refined, rf_y_pred_proba_refined],
model_names=["XGBoost", "Random Forest"]
)
/var/folders/sq/bfcsfb3x1ndczwvbpfn7f41c0000gn/T/ipykernel_55859/4204707867.py:10: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x="Importance", y="Feature", data=feature_importance, palette="viridis")
/var/folders/sq/bfcsfb3x1ndczwvbpfn7f41c0000gn/T/ipykernel_55859/4204707867.py:24: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x=fraud_rates.index, y=fraud_rates.values, palette="magma")
Comparison of ROC-AUC Curves for XGBoost and Random Forest¶
1. ROC-AUC Curve Overview¶
- The blue line represents the ROC curve for XGBoost, and the orange line represents the ROC curve for Random Forest.
- The dashed red diagonal line is the baseline, representing random guessing (AUC = 0.5).
2. Observations¶
XGBoost:
- Achieves an AUC of 1.000, indicating near-perfect performance.
- The ROC curve is very close to the top-left corner, reflecting excellent sensitivity (True Positive Rate) and specificity (low False Positive Rate) across thresholds.
Random Forest:
- Achieves an AUC of 0.997, which is also exceptional but slightly lower than XGBoost.
- The curve is slightly less steep than XGBoost's curve, particularly near the top-left corner, indicating slightly more False Positives or missed True Positives at certain thresholds.
3. Key Findings¶
Performance:
- Both models perform exceptionally well, with AUC values close to 1, making them highly effective for fraud detection.
- XGBoost marginally outperforms Random Forest, with slightly better discrimination between fraudulent and non-fraudulent transactions.
Generalization:
- The steeper curve for XGBoost suggests that it generalizes better across thresholds, maintaining higher True Positive Rates while minimizing False Positives.
4. Conclusion¶
- While both models are strong candidates, XGBoost outperforms Random Forest based on the ROC-AUC score and curve steepness.
- XGBoost is recommended as the final model for deployment due to its superior performance and robustness.
In [93]:
from catboost import CatBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score
# Check for categorical columns
categorical_columns = ["type", "State", "merchant_category", "time_of_day"]
existing_cat_features = [col for col in categorical_columns if col in X_train.columns]
# Train CatBoost model
catboost_model = CatBoostClassifier(
iterations=500,
learning_rate=0.1,
depth=6,
random_seed=42,
verbose=100,
cat_features=existing_cat_features # Only use existing categorical columns
)
catboost_model.fit(X_train, y_train)
# Predictions
catboost_y_pred = catboost_model.predict(X_test)
catboost_y_pred_proba = catboost_model.predict_proba(X_test)[:, 1]
Trying Catboost Model and SHAP Analysis
CatBoost (Categorical Boosting) is a gradient boosting library designed to handle categorical features efficiently and improve model performance. it is particularly well-suited for datasets with mixed data types (numerical and categorical)
In [99]:
import shap
import matplotlib.pyplot as plt
# Initialize SHAP explainer for CatBoost
explainer = shap.TreeExplainer(catboost_model)
# Calculate SHAP values for the test set
shap_values = explainer.shap_values(X_test)
# Step 1: Summary Plot
print("\nGenerating SHAP Summary Plot...")
shap.summary_plot(shap_values, X_test, plot_type="bar")
plt.show()
# Step 2: Detailed Force Plot for a Single Prediction
print("\nGenerating SHAP Force Plot for a Specific Prediction...")
sample_index = 0 # Change this index to analyze other predictions
shap.force_plot(
explainer.expected_value,
shap_values[sample_index],
X_test.iloc[sample_index],
matplotlib=True
)
# Step 3: Dependence Plot for a Key Feature
print("\nGenerating SHAP Dependence Plot for 'amount'...")
shap.dependence_plot("amount", shap_values, X_test)
Generating SHAP Summary Plot...
Generating SHAP Force Plot for a Specific Prediction...
Generating SHAP Dependence Plot for 'amount'...
In [105]:
import shap
import matplotlib.pyplot as plt
# Use CatBoost-specific SHAP TreeExplainer
print("\nInitializing SHAP Explainer for CatBoost...")
explainer = shap.TreeExplainer(catboost_model)
# Calculate SHAP values for the test set
print("\nCalculating SHAP Values...")
shap_values = explainer.shap_values(X_test)
# SHAP Summary Plot
print("\nGenerating SHAP Summary Plot...")
shap.summary_plot(shap_values, X_test, plot_type="bar")
# Visualize SHAP values for individual predictions
sample_index = 10 # Change this index to explain different samples
print(f"\nGenerating SHAP Force Plot for sample index {sample_index}...")
if isinstance(explainer.expected_value, list):
expected_value = explainer.expected_value[0]
else:
expected_value = explainer.expected_value
shap.force_plot(
expected_value,
shap_values[sample_index], # Use correct indexing for SHAP values
X_test.iloc[sample_index],
matplotlib=True,
)
# SHAP Dependence Plot for a specific feature
feature_to_plot = "num__amount" # Change this to another feature for different plots
print(f"\nGenerating SHAP Dependence Plot for feature: {feature_to_plot}...")
shap.dependence_plot(feature_to_plot, shap_values, X_test)
Initializing SHAP Explainer for CatBoost... Calculating SHAP Values... Generating SHAP Summary Plot...
Generating SHAP Force Plot for sample index 10...
Generating SHAP Dependence Plot for feature: num__amount...
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[105], line 35 33 feature_to_plot = "num__amount" # Change this to another feature for different plots 34 print(f"\nGenerating SHAP Dependence Plot for feature: {feature_to_plot}...") ---> 35 shap.dependence_plot(feature_to_plot, shap_values, X_test) File /opt/anaconda3/lib/python3.12/site-packages/shap/plots/_scatter.py:567, in dependence_legacy(ind, shap_values, features, feature_names, display_features, interaction_index, color, axis_color, cmap, dot_size, x_jitter, alpha, title, xmin, xmax, ax, show, ymin, ymax) 564 if len(features.shape) == 1: 565 features = np.reshape(features, (len(features), 1)) --> 567 ind = convert_name(ind, shap_values, feature_names) 569 # guess what other feature as the stongest interaction with the plotted feature 570 if not hasattr(ind, "__len__"): File /opt/anaconda3/lib/python3.12/site-packages/shap/utils/_general.py:46, in convert_name(ind, shap_values, input_names) 44 return "sum()" 45 else: ---> 46 raise ValueError("Could not find feature named: " + ind) 47 else: 48 return nzinds[0] ValueError: Could not find feature named: num__amount
SHAP Analysis with Force Plot and Dependence Plot¶
1. SHAP Force Plot¶
What It Represents:
- The SHAP force plot explains the contribution of each feature for a single prediction.
- Features in red push the prediction toward the positive class (e.g., fraud), while features in blue push it toward the negative class (non-fraud).
Key Observations:
- oldbalanceOrg and time_of_day_Night contribute significantly to pushing the prediction toward the positive class, as indicated by their large red bars.
- newbalanceOrig, type_CASH_OUT, and other blue features counteract the prediction, pushing it toward the negative class.
Baseline Value (f(x)):
- The model starts with a base prediction value (-15.19 in this case).
- The cumulative SHAP values adjust this base value, resulting in the final prediction score.
Inference:
- This graph helps identify which features strongly influenced the decision for this specific instance, providing transparency in the model's prediction process.
2. SHAP Dependence Plot¶
What It Represents:
- The SHAP dependence plot shows the relationship between the feature amount and its SHAP values, indicating how changes in amount influence the prediction.
- Points are color-coded based on another feature (type_CASH_OUT), capturing potential interactions.
Key Observations:
- Transactions with higher amounts (to the right of the x-axis) generally have negative SHAP values, indicating a reduced likelihood of fraud.
- Smaller transaction amounts (left side of the x-axis) show greater variability in SHAP values, with some pushing predictions toward fraud (positive SHAP values).
- The color coding reveals that type_CASH_OUT significantly interacts with amount, with red points showing higher SHAP values (fraud indicators).
General SHAP Insights¶
Transparency:
- These plots enhance interpretability, helping identify how individual features and their interactions contribute to predictions.
Actionable Insights:
- Features like time_of_day_Night and transaction types (type_CASH_OUT) play critical roles in fraud detection.
- Understanding interactions between features (e.g., amount and type_CASH_OUT) can guide feature engineering or business rules.
Fraud Detection Applications:
- For smaller amounts or night-time transactions, higher SHAP values suggest the need for additional scrutiny, as these patterns are more indicative of fraud.
In [107]:
# Print all available feature names in X_test
print("\nAvailable Features in X_test:")
print(X_test.columns)
# Update the feature_to_plot with a valid feature name
feature_to_plot = "amount" # Replace this with the correct column name from the printed list
print(f"\nGenerating SHAP Dependence Plot for feature: {feature_to_plot}...")
# Generate the dependence plot with the corrected feature name
shap.dependence_plot(feature_to_plot, shap_values, X_test)
Available Features in X_test:
Index(['amount', 'oldbalanceOrg', 'newbalanceOrig', 'isFlaggedFraud', 'step',
'newbalanceDest', 'oldbalanceDest', 'type_CASH_OUT', 'type_DEBIT',
'type_PAYMENT', 'type_TRANSFER', 'time_of_day_Evening',
'time_of_day_Morning', 'time_of_day_Night', 'State_Alaska',
'State_Arizona', 'State_Arkansas', 'State_California', 'State_Colorado',
'State_Connecticut', 'State_Delaware', 'State_Florida', 'State_Georgia',
'State_Hawaii', 'State_Idaho', 'State_Illinois', 'State_Indiana',
'State_Iowa', 'State_Kansas', 'State_Kentucky', 'State_Louisiana',
'State_Maine', 'State_Maryland', 'State_Massachusetts',
'State_Michigan', 'State_Minnesota', 'State_Mississippi',
'State_Missouri', 'State_Montana', 'State_Nebraska', 'State_Nevada',
'State_New Hampshire', 'State_New Jersey', 'State_New Mexico',
'State_New York', 'State_North Carolina', 'State_North Dakota',
'State_Ohio', 'State_Oklahoma', 'State_Oregon', 'State_Pennsylvania',
'State_Rhode Island', 'State_South Carolina', 'State_South Dakota',
'State_Tennessee', 'State_Texas', 'State_Utah', 'State_Vermont',
'State_Virginia', 'State_Washington', 'State_West Virginia',
'State_Wisconsin', 'State_Wyoming'],
dtype='object')
Generating SHAP Dependence Plot for feature: amount...
In [111]:
print("SHAP values shape:", shap_values.shape)
print("X_test shape:", X_test.shape)
SHAP values shape: (263111, 63) X_test shape: (263111, 63)
In [113]:
print("Unique values in 'amount':", X_test['amount'].unique())
Unique values in 'amount': [ 7443.43 10000.24 520937.07 ... 3284.91 73197.48 78954.1 ]
In [117]:
# SHAP Summary Plot
print("Generating SHAP Summary Plot...")
shap.summary_plot(shap_values, X_test)
Generating SHAP Summary Plot...
General Inferences¶
Feature Importance:
- The ranking of features highlights the critical variables for fraud detection, with transaction balances, amounts, and times dominating the model's decision-making.
Interaction Insights:
- The distribution of red and blue points along the x-axis reveals how different feature values interact with SHAP values:
- For example, low values of balances (
newbalanceOrig,oldbalanceOrg) push predictions toward fraud, while high values often reduce fraud likelihood.
- For example, low values of balances (
- The distribution of red and blue points along the x-axis reveals how different feature values interact with SHAP values:
Variability Across Features:
- The spread of points along the x-axis for each feature indicates the variability in its impact across different predictions.
Why This Is Useful¶
Interpretability:
- The plot makes it clear how individual features influence the model’s predictions, bridging the gap between complex machine learning models and human understanding.
Feature Engineering:
- Identifying highly impactful features can guide efforts to refine or add new variables for improving model performance.
In [118]:
# SHAP Force Plot for Local Interpretations
sample_index = 10 # Choose a specific test sample to interpret
print(f"\nGenerating SHAP Force Plot for sample index {sample_index}...")
shap.force_plot(
base_value=explainer.expected_value,
shap_values=shap_values[sample_index],
features=X_test.iloc[sample_index],
feature_names=X_test.columns,
matplotlib=True
)
Generating SHAP Force Plot for sample index 10...
In [121]:
# SHAP Summary Plot
print("Generating SHAP Summary Plot...")
shap.summary_plot(shap_values, X_test)
Generating SHAP Summary Plot...
In [122]:
# SHAP Force Plot for Local Interpretations
sample_index = 10 # Change this index to analyze different samples
print(f"\nGenerating SHAP Force Plot for sample index {sample_index}...")
shap.force_plot(
base_value=explainer.expected_value,
shap_values=shap_values[sample_index],
features=X_test.iloc[sample_index],
feature_names=X_test.columns,
matplotlib=True
)
Generating SHAP Force Plot for sample index 10...
In [125]:
# SHAP Dependence Plot
feature_to_plot = "amount" # Replace with the feature name you want to analyze
print(f"\nGenerating SHAP Dependence Plot for feature: {feature_to_plot}...")
shap.dependence_plot(feature_to_plot, shap_values, X_test)
Generating SHAP Dependence Plot for feature: amount...
In [127]:
# Calculate mean absolute SHAP values for each feature
mean_abs_shap_values = pd.DataFrame({
'Feature': X_test.columns,
'Mean_Absolute_SHAP_Value': np.abs(shap_values).mean(axis=0)
}).sort_values(by='Mean_Absolute_SHAP_Value', ascending=False)
# Top 10 important features
print("\nTop 10 Features by SHAP Value (Global Importance):")
print(mean_abs_shap_values.head(10))
Top 10 Features by SHAP Value (Global Importance):
Feature Mean_Absolute_SHAP_Value
2 newbalanceOrig 3.646211
1 oldbalanceOrg 2.203511
0 amount 1.255044
4 step 0.839119
7 type_CASH_OUT 0.742726
5 newbalanceDest 0.657398
6 oldbalanceDest 0.631414
13 time_of_day_Night 0.447159
10 type_TRANSFER 0.216774
9 type_PAYMENT 0.055400
In [129]:
# Select a sample index to analyze
sample_index = 10 # Change this index to analyze another sample
sample_shap_values = shap_values[sample_index]
# Create a DataFrame to display feature contributions
local_shap_df = pd.DataFrame({
'Feature': X_test.columns,
'SHAP_Value': sample_shap_values
}).sort_values(by='SHAP_Value', ascending=False)
print(f"\nSHAP Values for Prediction at Index {sample_index}:")
print(local_shap_df.head(10))
SHAP Values for Prediction at Index 10:
Feature SHAP_Value
1 oldbalanceOrg 2.750442
0 amount 0.300984
11 time_of_day_Evening 0.057501
9 type_PAYMENT 0.008553
25 State_Illinois 0.007259
26 State_Indiana 0.003761
35 State_Minnesota 0.003724
55 State_Texas 0.003199
47 State_Ohio 0.001612
61 State_Wisconsin 0.001092
Table Description¶
- This table ranks the top 10 features by their mean absolute SHAP value, which quantifies their overall importance in influencing the model's predictions.
- The higher the Mean Absolute SHAP Value, the greater the average contribution of that feature to the model's decisions.
In [131]:
# Summarize SHAP values for a specific feature
feature_to_analyze = "amount" # Change this to analyze another feature
feature_shap_values = shap_values[:, X_test.columns.get_loc(feature_to_analyze)]
print(f"\nDependence Summary for Feature: {feature_to_analyze}")
print(f"Mean SHAP Value: {np.mean(feature_shap_values)}")
print(f"Median SHAP Value: {np.median(feature_shap_values)}")
print(f"Min SHAP Value: {np.min(feature_shap_values)}")
print(f"Max SHAP Value: {np.max(feature_shap_values)}")
Dependence Summary for Feature: amount Mean SHAP Value: -0.1590104647438564 Median SHAP Value: 0.09935050738430434 Min SHAP Value: -8.590958576991001 Max SHAP Value: 10.4961963813414
Interpretation and Results¶
Project Overview¶
In this analysis, we explored fraud detection using various machine learning models, evaluated their performance, and interpreted the results to understand the significance of features and patterns in fraudulent transactions. The primary goal was to identify critical fraud indicators and build a reliable model to predict fraudulent transactions accurately.
Model Insights¶
XGBoost Model:
- Achieved the highest accuracy and ROC-AUC score (1.000), indicating excellent performance and strong discriminatory power.
- Performed well in balancing both Type 1 errors (false positives) and Type 2 errors (false negatives), making it the most suitable model for deployment.
Random Forest Model:
- Performed slightly below XGBoost, with an ROC-AUC score of 0.997.
- Demonstrated strong overall accuracy and precision but missed some fraudulent transactions compared to XGBoost.
Logistic Regression:
- Struggled to capture the complexity of the dataset, with lower accuracy compared to tree-based models.
- Highlighted the importance of non-linear models for this problem.
Feature Importance and Patterns¶
Behavioral Patterns:
- Fraudulent transactions were more likely to occur at night, with lower account balances and specific transaction types being strong indicators.
Feature Interactions:
- Interaction between balances, transaction amounts, and types played a significant role in shaping predictions, as highlighted by SHAP analysis.
Evaluation of Current Models¶
Model Performance:
- While XGBoost outperformed other models, the marginal difference in performance with Random Forest suggests both models are effective for this dataset.
- Logistic Regression highlighted the limitations of linear models for highly imbalanced and complex datasets.
Parameter Tuning:
- Grid Search CV did not yield significant improvements, suggesting that the current feature set might not fully capture the complexity of fraud detection.
Key Observations¶
- The existing features provide valuable insights but might not be sufficient for further performance gains.
- SHAP analysis revealed the transparency of the models, aligning their predictions with domain knowledge and emphasizing the importance of interpretability.
SHAP Analysis¶
- SHAP visualizations provided global and local interpretability for the models:
- Summary Plot: Ranked features by their global importance, confirming the significance of balance, amount, and transaction type features.
- Dependence Plot: Showed strong interactions between
amountand categorical features liketype_CASH_OUT. - Force Plot: Explained individual predictions, offering transparency in the decision-making process.
Final Takeaway¶
This project provided a robust understanding of fraudulent transaction patterns and effective modeling techniques. XGBoost stood out as the optimal model, offering both high accuracy and interpretability. While current models perform well, incorporating more features and iterating through the lifecycle will enhance predictions and provide actionable insights for fraud prevention strategies.
References¶
data- https://www.kaggle.com/datasets/rupakroy/online-payments-fraud-detection-dataset
book- Hands-on Machine Learning with Scikit Learn Tensorflow and Keras O'Riely
Research Paper - A Unified Approach to Interpreting Model Predictions https://arxiv.org/abs/1705.07874