Data Analytics

sns.lineplot(data=summary, x="Cutoff", y="Error rate")

ax = summary.plot(x="Cutoff", y="Accuracy", legend=False)

ax2 = ax.twinx() summary.plot(x="Cutoff", y="Error rate",

ax=ax2, legend=True, color="r")

toyota_df = pd.read_csv('ToyotaCorolla.csv', encoding="ISO-8859-1") toyota_df.head()

toyota_df[["Fuel_Type", "Price"]]

toyota_df.Fuel_Type.value_counts() banking_df.Mortgage.value_counts().sort_index()

toyota_df.groupby('Fuel_Type').Price.mean()

sns.boxplot(x="Fuel_Type", y="Price", data=toyota_df, whis=100)

with pd.option_context('mode.use_inf_as_na', True): sns.set(rc={'figure.figsize':(13,5), "figure.dpi":300,})

sns.set_theme(style="whitegrid")sns.swarmplot(x="Fuel_Type", y="Price", data=toyota_df, size=4)

with pd.option_context('mode.use_inf_as_na', True): sns.set(rc={'figure.figsize':(10,8), "figure.dpi":300,})

sns.set_theme(style="whitegrid")sns.stripplot(x="Fuel_Type", y="Price", data=toyota_df)

modg_X = toyota_df[['Fuel_Type']

]modg_X = pd.get_dummies(modg_X, drop_first=True)

modg_X = sm.add_constant(modg_X)

modg_X = modg_X.astype(float) # Make sure that all columns have numerical values# Model estimation and results

modg = sm.OLS(toyota_df['Price'], modg_X)res = modg.fit()print(res.summary())

# Fuel_Type transform in Dummies

X = toyota_df[['Fuel_Type', 'HP']]

y = toyota_df[['Price']]# Transform Fuel_Type in dummies

X = pd.get_dummies(X, drop_first=True)# Split the datatrain_X, valid_X, train_y,

valid_y = train_test_split(X, y, test_size=0.4)# Model

fittingtoyota_ml = LinearRegression()toyota_ml.fit(train_X, train_y)

print('Performance Measures (Training data)') regressionSummary(train_y, toyota_ml.predict(train_X))

print('Performance Measures (Validation data)') regressionSummary(valid_y, toyota_ml.predict(valid_X))

banking_df.columns = [s.strip().replace(" ", "_") for s in banking_df.columns] banking_df.head()

banking_df["Education"].value_counts().sort_index()

banking_df["Education"] = banking_df["Education"].map({1: "Undergrad", 2: "Graduate", 3: "Advanced/Professional"})

banking_df.head()

banking_df["has_mortgage"] = [0 if x == 0 else 1 for x in banking_df["Mortgage"]]

banking_df.head()

X_simple = banking_df["Income"]

Y_simple = banking_df["has_mortgage"]

X_simple = sm.add_constant

(X_simple)logit_simple_mod = sm.Logit

(Y_simple, X_simple)logit_simple_mod_res = logit_simple_mod.fit()print(logit_simple_mod_res.summary())

X_full = banking_df[["Income", "Family", "CCAvg", "Education", "Age"]] X_full = pd.get_dummies(X_full, prefix_sep="_", drop_first=True)

X_full = X_full.astype(float) # Make sure that all columns have numerical data types

Y_full = banking_df["has_mortgage"] X_full = sm.add_constant

(X_full)logit_full_mod = sm.Logit(Y_full, X_full)

logit_full_mod_res = logit_full_mod.fit()print(logit_full_mod_res.summary())

predict_valid = logit_reg.predict(valid_X) cm2 = confusion_matrix(valid_y, predict_valid)

ConfusionMatrixDisplay(cm2).plot()

import kds as kds

kds.metrics.plot_lift(valid_y, predict_valid)

Ordinal -> values can be ordered logically (e.g. good - ok - bad) and

nominal -> values cannot be ordered logically (e.g. blue, yellow, red)

Data Analysis, Visualisation, Prediction, Classification, Data Reduction, Association Rules, Recommendation Systems

Define/ Understand the purpose of the analysis;

Obtaining data (possibly including sampling);

Data analysis, cleaning, preparation;

Reduce the data (dimension) (for supervised data mining, partition it);

Specify the analysis goal (classification, prediction, etc.);

Selection of techniques (e.g. regression, logit);

Iterative implementation and tuning;

Evaluation of the results;

Roll-out and widespread use of the best model

Supervised learning's goal is to predict a target or outcome variable; the goal of unsupervised learning is to identify patterns and divide data into meaningful groups.

In supervised learning a target value is known in the training data, it is the data on which the algorithm is trained; in unsupervised learning there is no target variable for prediction or classification.

Supervised learning: classification and prediction; unsupervised learning: association rules, data reduction, data exploration, visualisation.

The goal is the prediction of a numerical outcome variable.

The goal is the prediction of a categorical outcome variable.

housing_df.describe()

housing_df.shape

housing_df.head()

Number of categories - 1 -> redundant information leads to the failure of algorithms.

Observation that is extreme (far away) compared to the rest of the data.

We need expertise in the data to determine whether it is an error or a true extreme. Sometimes it is possible to correct the error. If the number of outliers is small and we recognize it as an error: treat it as a missing value.

Graphically, ordering the variables, using the minimum/maximum values.

If no data value is stored for a variable for an observation; denoted with NaN.

Solution 1: Discard - only practical if the number of missing records is small. Solution 2: Imputation - replace the missing values with meaningful substitute values (mean, median) -> advantage: we can keep the observation’s non-missing information.

housing_df["BEDROOMS"].sort_values()

Standardisation puts all variables on the same scale.