Data Analytics Wissen

zi = (xi - x)/s. Subtract the arithmetic mean and divide by the standard deviation. Alternative: Rescale - subtract the minimum and divide by the range of max and min.

Standardisation puts all variables on the same scale.

housing_df["BEDROOMS"].sort_values()

Solution 1: Discard - only practical if the number of missing records is small. Solution 2: Imputation - replace the missing values with meaningful substitute values (mean, median) -> advantage: we can keep the observation’s non-missing information.

If no data value is stored for a variable for an observation; denoted with NaN.

Graphically, ordering the variables, using the minimum/maximum values.

We need expertise in the data to determine whether it is an error or a true extreme. Sometimes it is possible to correct the error. If the number of outliers is small and we recognize it as an error: treat it as a missing value.

Observation that is extreme (far away) compared to the rest of the data.

Number of categories - 1 -> redundant information leads to the failure of algorithms.

The goal is the prediction of a categorical outcome variable.

The goal is the prediction of a numerical outcome variable.

Supervised learning: classification and prediction; unsupervised learning: association rules, data reduction, data exploration, visualisation.

In supervised learning a target value is known in the training data, it is the data on which the algorithm is trained; in unsupervised learning there is no target variable for prediction or classification.

Supervised learning's goal is to predict a target or outcome variable; the goal of unsupervised learning is to identify patterns and divide data into meaningful groups.

Define/ Understand the purpose of the analysis;

Obtaining data (possibly including sampling);

Data analysis, cleaning, preparation;

Reduce the data (dimension) (for supervised data mining, partition it);

Specify the analysis goal (classification, prediction, etc.);

Selection of techniques (e.g. regression, logit);

Iterative implementation and tuning;

Evaluation of the results;

Roll-out and widespread use of the best model

Data Analysis, Visualisation, Prediction, Classification, Data Reduction, Association Rules, Recommendation Systems

Ordinal -> values can be ordered logically (e.g. good - ok - bad) and

nominal -> values cannot be ordered logically (e.g. blue, yellow, red)

Continuous -> infinite numbers (e.g. size, time, age) and

Integer -> integer (e.g. number of cars, number of cities)