Jonas Wagner machine_learning_and_malware.pdf

1. Collect examples (e.g. SPAM and not-SPAM mails) . These examples are used to train the machine learning system.
2. Extract features from each training example to represent the ecample as an array of numbers. This step also includes research to design good features that will help a machine learning system make accurate inferences.
3. Train the machine learning system using the features we have extracted from e.g. recognizing SPAM or not-SPAM mails.
4. Test the system on some examples not included in the training examples.

• Quality of data is very important
  • Data scientists spend a lot of time collectiong and cleaning up data
  • The better and more divers the data, the better the model
• If the data does not contain certain cases, that model won't recognize it, e.g. can't detect cats in pictures if the training data doesn't contain pictures of cats.
• Needs to have labels
  • This is a huge problem in IT-Security e.g. have 1mio binaries to train good/bad clasifier -> how do you know which ones are actually good or bad?

• A macine learning algorithm needs features of examples to work with, for a good/bad classifiert on files this might be:
  • Is it digitally signed?
  • Are the file headers malformed?
  • Is the entropy of the file high? (could indicate encryption)
• Challenge: Needs a lot of domain knowledge
  • What are good featurs?
  • Is every feature of the same importance
  • How do I hanle a large amount of features

• based on the extracted features (and optionally labels), a model can be trained with an algorithm.
• The choice of the algorithm is always based on what you're trying to achive.

• After training the quality of the model is tested with test examples
• Usually done by splitting collected data into training (80%) and test (20%) set. If not, then you will run into over or underfitting.

Underfitet -> doesn't match the data set well bad for detection

Overfittet -> matches to exactly will not detect other samples that are simular.

• split dataset into k parts, take k tests with different training and test (or validation) sets and average the scores of all tests

• This pipeline usually gets repeated many times until a model with good accuracy is found.
• The world keeps changing, e.g. new attacker data appears frequently, which means the model needs to be retrained with new data.

Machine learning: feature exrtraction is done by humans

Deep learning: everything is done by machines.

• Classification of PE files
• Detecting malicious code loading in processes
• Detect code simularities