IAI | HSLU | Magdalena Picariello

Yes, you can make progress without big data.

Having more data never hurts and can often enhance AI performance.

Gathering large volumes of data can be very expensive and resource-intensive.

Yes, you may be able to bring value to your AI project even with the limited data that you have. It depends on the specific project's goals and requirements.

●  There is no obligation to publish code, model, and data

●  If model (inference code) is published, it doesn’t mean that training code is published too

●  The exact training datasets are rarely available

●  Even if they are, the preprocessing code may be missing

●  Code quality is often poor (not compiling, missing dependencies etc.)

●  Described models are tweaked to some metric

Echo / Alexa

1. Collect data

   • ○  Collect audio clips of people saying “Alexa”

   • ○  Collect audio clips of people saying other stuff

2. Train model

   • ○  Classify audio clips (Alexa/Not Alexa)

   • ○  Iterate many times till good enough

3. Deploy model

   • ○  Put ML software in the smart speaker

   • ○  Get data back for failing cases

   • ○  Maintain/update model

Optimizing a sales funnel

1. Collect data
○ History of users going to web pages (time/country/website)

2. Analyze data
○ Iterate many times to get good insights

3. Suggest hypotheses/actions

• ○  Deploy changes

• ○  Re-analyze data periodically

1. Collect data
2. Train model: iterate

● Analyze data (actual vs expected)

● Formulate hypothesis
● Suggest actions
● Implement, test, and repeat

3. Deploy the model

The initial phase in the machine learning project lifecycle is Scoping, where the project's objectives and requirements are defined.

In the "Data" phase, the activities include defining data, establishing a baseline, labeling and organizing data.

The "Modeling" phase involves selecting and training a model for the machine learning projec as well as Perform error analysis.

• Productize the model
• Monitor and maintain the system. 

• -  Defined for you

• -  Goal: due diligence on it

  • -  Is it feasible?

  • -  What resources do I need?

• -  What metric do I use?

VS:

• -  Provided

• -  You need to do due diligence

• -  Do I have enough features?

• -  Do you need additional data sources?

• -  Is the data quality good enough?

VS:

• -  Out of scope

• -  Data drift/concept drift

• -  Data and Deployment is where most of the work happens

• -  Very little jobs for the modeling

1. Establish the baseline level of performance

2. Implement quick-and-dirty model

3. Perform sanity checks

4. Error analysis

• ●  Human level performance

• ●  Literature search for state-of-art/open source

• ●  Quick-and-dirty implementation

• ●  Performance of older system

  Baseline helps us understand what may be possible.
  In some cases (e.g. HLP) it also gives us understanding of the irreducible error.

●  Do literature search to see what is possible (kaggle forum, courses, blogs, open-source-project)

●  Find open source implementations if available

A reasonable algorithm with good data will often outperform a great algorithm with not so good data.

We are only humans, and our code is error prone.

● Get a small subset of the data
● Train an overfitting model
● Does your model deliver expected results?

● Are your results uploadable to Kaggle?

●  One number

●  Can capture few aspects

●  Evaluated at different stages of the process

●  Allows you to track progress

Evaluation metric allows you to quickly tell if model A or model B is better.
Having a dev set plus single number evaluation metric allows to speed up iterating.

As long as ML is worse than humans, you can:

●  Get labeled data from humans

●  Gain insights from manual error analysis: why did person get it right?

●  Do better analysis of bias/variance

●  How much room for improvement there is

●  How frequently that category appears

●  How easy it is to improve accuracy in that category

●  How important it is to improve this category