IAI | HSLU | Magdalena Picariello

It involves randomly sampling hyperparameter values to avoid potential biases and to ensure a thorough search.

It refers to starting with a wide exploration of hyperparameter values (coarse) and gradually narrowing down the search to find the optimal values (fine).

Create realistic examples that (1) algorithm does poorly on, but (2) humans do well on

Data augmentation aims to generate additional training examples to enhance the performance of a machine learning model.

●  Does it sound realistic?

●  Is the x->y mapping clear (e.g. can humans recognize speech?)

●  Is the algorithm currently doing poorly on it?

With unstructured data, we can perform artificial data synthesis

With structured data, we can do feature engineering.
 

Example: Restaurant recommender
Vegetarians are frequently recommended restaurants with only meat options Possible features to add:

• ●  Is person vegetarian (based on past orders)?

• ●  Does restaurant have vegetarian options (based on menu)

Data iteration involves continuously revising and improving features based on the results of error analysis, user feedback, and benchmarking. It helps enhance the quality and relevance of features.

• Error analysis becomes challenging without a good baseline (Human-Level Performance or HLP) because you lack a reliable reference point for comparison. A baseline provides insight into how well a human can perform the task, which is essential for identifying areas where the model falls short.

A1: User feedback is valuable for identifying which features users find relevant or missing. It provides insights that guide the selection and engineering of features to enhance user satisfaction and model performance.

A2:  Benchmarking against competitors helps identify areas where your model can gain a competitive edge. By analyzing competitor performance, you can inspire the development of unique features that set your model apart in the field.

●  Noisy background:

   ○  Cafe noise

   ○  Car noise

●  Accented speech

●  Far from microphone

●  Young children

●  Stuttering

AI system = Code (model) + data

• Traditional machine learning research driven by improving benchmark dataset performance.
• Researchers often work on a fixed dataset they download.
• This approach has led to significant progress in machine learning.
• In production systems, the dataset doesn't need to remain fixed.
• It's common to edit the training and test sets to enhance data quality for better system performance.

• Think about automating tasks rather than automizing jobs, e.g. call center routing, radiologists
• What are the main drives of the business value?
• What are the main pain points in the business?

1. ●  Can we meet desired performance?

2. ●  Can we use pre-existing components?

3. ●  How much data is needed?

4. ●  What resources are needed?

5. ●  What are the dependencies?

6. ●  Are there any legal constraints?

● Does it lower costs?
● Does it generate revenue?
● Does it enable launching new product?
● Does it generate ENOUGH value?

1. Do other people solve similar problems?

2. What performance do they achieve?

3. With my skills, what performance can I achieve?

4. Do I need additional resources?

5. Can I do it in a reasonable time?

Yes, you can make progress without big data.

Having more data never hurts and can often enhance AI performance.

Gathering large volumes of data can be very expensive and resource-intensive.

Yes, you may be able to bring value to your AI project even with the limited data that you have. It depends on the specific project's goals and requirements.

●  There is no obligation to publish code, model, and data

●  If model (inference code) is published, it doesn’t mean that training code is published too

●  The exact training datasets are rarely available

●  Even if they are, the preprocessing code may be missing

●  Code quality is often poor (not compiling, missing dependencies etc.)

●  Described models are tweaked to some metric

Echo / Alexa

1. Collect data

   • ○  Collect audio clips of people saying “Alexa”

   • ○  Collect audio clips of people saying other stuff

2. Train model

   • ○  Classify audio clips (Alexa/Not Alexa)

   • ○  Iterate many times till good enough

3. Deploy model

   • ○  Put ML software in the smart speaker

   • ○  Get data back for failing cases

   • ○  Maintain/update model