IAI | HSLU | Magdalena Picariello

1. Fit training set well on cost function

2. Fit dev set well on cost function

3.Fit test set well on cost function

4. Performs well in real world

Different distributions can lead to issues because you may end up optimizing your model for the dev set and achieving poor performance on the test set, which is the true measure of your model's generalization.

Choose a dev set and test set to reflect data you expect to get in the future and consider important to do well.

Mismatches between train/test distributions are often unavoidable, but ensuring that dev/test sets have the same distributions can lead to better long-term performance. This practice helps align the model evaluation with real-world scenarios, leading to more robust and reliable results.

• Try using a bigger neural network to increase model capacity.
• Experiment with other optimization algorithms to improve convergence.

• Apply regularization techniques to prevent overfitting.
• Increase the training set size to enhance model learning.

• Consider using a bigger dev set to ensure it represents the target distribution.

• Evaluate if a change in the dev set is required to make it more representative of real-world data.
• Investigate the possibility of modifying the cost function to better align the model with the desired task.

Error = Bias^2 + Variance + Niose

High variance suggests a model is overfitting the training data, resulting in a large gap between train set and dev set errors.

low train error + high dev error + big gap 

High bias suggests a model is underfitting the data, causing small gaps between train set and dev set errors, but both have high error rates.

high train error + high dev error + small gap 

When a model has both high bias and high variance, it means it both oversimplifies and overcomplicates the problem, resulting in high errors and big gap.

High train error + high dev error + high gap

The ideal scenario is low bias and low variance, where the model finds a balance between complexity and generalization, leading to accurate predictions.

Low train and low dev errors + small gap

-  Bigger neural network

-  Train longer

-  (NN architecture search)

-  More data

-  Regularization

-  (NN architecture search)

Focus on Bias

• A strategy to address high bias, often related to underfitting, by improving the model's capacity and complexity.

Focus on Variance

• A strategy to address high variance, often related to overfitting, by reducing the model's complexity and regularization.

Avoidable Bias

• Error that can be reduced through improved algorithms, more data, or better features.

Variance

• The model's sensitivity to small fluctuations or noise in the data.

Look at dev examples to evaluate ideas Error analysis:

●  Get ~ 100 mislabeled dev set examples

●  Count up how many are muffins


 

Improvement Ceiling

• The maximum potential improvement that can be achieved by addressing the identified mislabeled examples.

In Scenario 1:

• 50 out of 100 mislabeled examples were muffins.
• The model error is 10%.
• The improvement ceiling is calculated as 50% * 10% = 5%.

In Scenario 2:

• 5 out of 100 mislabeled examples were muffins.
• The model error is 10%.
• The improvement ceiling is calculated as 5% * 10% = 0.5%.

The main goal is to determine whether the idea is worth pursuing further or investing effort into.

You should consider evaluating a single idea in your machine learning project if the following criteria are met:

• Your algorithm is not performing at Human Level Performance.
• You have a specific idea or change to test.
• You can manually examine the mistakes your algorithm is making to assess the idea's potential impact.

A1: The purpose is to identify and prioritize ideas for enhancing the model's performance.

A2: The "Gap to HLP" represents the difference between the model's accuracy and human-level performance. It is crucial because it indicates how close the model is to achieving human-level performance.

A3: The "Improvement ceiling" represents the maximum potential improvement that can be achieved for a specific category. It is calculated by multiplying the gap to human-level performance by the percentage of data in that category.

A4: Evaluating the performance gap in different categories allows for a focused approach to improving the model's accuracy in areas where it lags behind human-level performance.

●  How much room for improvement there is

●  How frequently that category appears

●  How easy it is to improve accuracy in that category

●  How important it is to improve this category

The approach involves exploring hyperparameter choices by trying different values in a systematic manner.

Exploring a broad range of hyperparameter values is crucial to find the settings that optimize the model's performance.

The sequence in which hyperparameters are tuned can impact the final result, as adjustments can interact with each other.

It involves randomly sampling hyperparameter values to avoid potential biases and to ensure a thorough search.

It refers to starting with a wide exploration of hyperparameter values (coarse) and gradually narrowing down the search to find the optimal values (fine).

Create realistic examples that (1) algorithm does poorly on, but (2) humans do well on

Data augmentation aims to generate additional training examples to enhance the performance of a machine learning model.