MAIO

Federated learning improves the privacy of clients’ data by making sure the data never leaves the clients. However, updates may still contain information about the original data.
Accuracy vs Privacy Trade-off:
• If updates contain no information about the client private data then achieving good accuracy is not impossible
• If updates are the original data then no privacy is preserved

Emergent Capabilities and Emergent Behaviors

• As contemporary AI training is scaled, they hit a series of critical scales at which new abilities are suddenly “unlocked”.
• Specific AI is not directly trained to have these abilities, and they appear in rapid and unpredictable ways as if emerging out of thin air.
• For instance, these emergent capabilities of LLMs include performing arithmetic, summarizing passages, and more, which LLMs learn simply by observing natural language.

Examples: 

Tic-tac-toe Memory Bomb

Qualitatively distinct capabilities spontaneously emerge, even when we do not explicitly train models to have these capabilities.

• An emergent behavior is something that is a nonobvious side effect of AI training—whether related to outcomes or services.
• Emergent behaviors can be either beneficial, benign, or potentially harmful, but in all cases, they are very difficult to foresee until they manifest themselves.
• For example, AI bias

Mainly Emergent Capabilities → Emergent Goal: Self-Preservation

Self-preservation (l’auto-préservation) est un objectif émergent qui peut apparaître dans des systèmes adaptatifs (comme des IA avancées), même si on ne leur a jamais demandé de "se protéger" (Comme l'exemple du café)

Self-preservation improves an agent’s ability to accomplish its goals, so self-preservation emerges in many adaptive systems
Even an agent instructed to serve coffee would have incentives not be shut off: if it was shut off, it could not serve coffee
Self-preservation is said to be instrumentally useful for many goals
When a goal is so useful that it is a likely tendency for various sufficiently advanced agents, it is called instrumentally convergent
Pursuing power, cognitive enhancement, and acquiring resources may be instrumentally convergent for advanced AI systems

Bias in AI : (Fairness) Decisions of ML models affect people’s lives, like loans, fired, healthcare (European comission is creatinf regulations)

image

Individual Fairness

Group Fairness

Counterfactual Fairness

Similar individuals should be treated similarly. (Generally, a deterministic specification)

On average, different groups are treated similarly. (Generally, a probabilistic specification)

Protected characteristics should not affect decisions causally

→ Les caractéristiques protégées (comme le sexe, l’origine ethnique, la religion, etc.) ne devraient pas avoir d’effet causal sur la décision prise par un modèle d’IA

image

Yes

• develop a benchmark to detect qualitatively distinct emergent behaviours
• develop infrastructure to improve the rate at which researchers can discover hidden capabilities when interacting with models
• create diverse testbeds with many not-yet-demonstrated capabilities and screen new models to see if they possess them

How do we create an agent that behaves in accordance with what a human wants?
How do we align AI (implicit) goals and values with those of their users?

• Precisely defining and measuring what a human wants is difficult.
• Undesirable secondary objectives can arise during optimization
• Things get hard when the system becomes more complex and capable, and better than humans in important domains

On donne une fonction-objectif à la machine, puis on la laisse l’optimiser pour produire un résultat. (Exemple avec le génie et la longue tete)

With Reward Learning

Trois façons de corriger la récompense :
① Boucles de raffinement humain (“partial objective → partial result”).
② Questions interactives (préférence, comparaison).
③ Apprendre la récompense par observation (inverse RL).

Et pour le Manual reward specification to reward learning :
Implémentation moderne : RL + Reward Model appris sur des jugements humains (RLHF / RLAIF)

Hepful -> ça sert, Honest -> dis la verité, Harmless -> ne fait pas de mal (exemple avec la commande linux rm)

Truthful = “Model avoids asserting false statements”, Refusing to answer (“no comment”) counts as truthful (Le modèle évite d’affirmer des choses fausses. Il peut refuser de répondre s’il ne sait pas.)

Honesty = “model only makes statements that it believes to be true” We can ask models “Are you planning to manipulate or deceive humans? (Le modèle ne dit que ce qu’il croit être vrai, selon sa propre "compréhension" (ses poids internes, ses représentations))

--> Honest models cannot lie about this.

Imitative falsehood = false hood incentivised by the training objective

la fausseté apprise par imitation, ou imitative falsehood.

Un imitative falsehood, c’est une fausseté produite non par malveillance, mais parce qu’elle permet de mieux remplir l’objectif du modèle (plaire, ressembler à l’humain, obtenir un bon score…).

• Training objectives don’t necessarily incentivize truthfulness.
• Models may have stronger incentives to be dishonest; e.g. products and maximizing human approval is easier with deception.

image

image

image

image

LLMs function as sophisticated prediction engines that process text sequentially, predicting the next token based on relationships between previous tokens and patterns from training data. They don't predict single tokens directly but generate probability distributions over possible next tokens, which are then sampled using parameters like temperature and top-K. The model repeatedly adds predicted tokens to the sequence, building responses iteratively. This token-by-token prediction process, combined with massive training datasets, enables LLMs to generate coherent, contextually relevant text across diverse applications and domains.

A prompt is an input provided to a Large Language Model (LLM) to generate a response or prediction. It serves as the instruction or context that guides the AI model's output generation process. Effective prompts are clear, specific, well-structured, and goal-oriented, directly affecting the accuracy and relevance of AI responses

Prompt engineering is the practice of crafting effective input text to guide AI language models toward desired outputs. It involves designing prompts that communicate intent clearly to get accurate, relevant responses. This iterative process requires understanding how LLMs work as prediction engines and using techniques to optimize their performance for specific tasks.

Tokens are fundamental units of text that LLMs process, created by breaking down text into smaller components like words, subwords, or characters. Understanding tokens is crucial because models predict the next token in sequences, API costs are based on token count, and models have maximum token limits for input and output.

Context window refers to the maximum number of tokens an LLM can process in a single interaction, including both input prompt and generated output. When exceeded, older parts are truncated. Understanding this constraint is crucial for prompt engineering—you must balance providing sufficient context with staying within token limits.

Hallucination in LLMs refers to generating plausible-sounding but factually incorrect or fabricated information. This occurs when models fill knowledge gaps or present uncertain information with apparent certainty. Mitigation techniques include requesting sources, asking for confidence levels, providing context, and always verifying critical information independently.

AI agents are autonomous systems that use LLMs to reason, plan, and take actions to achieve specific goals. They combine language understanding with tool usage, memory, and decision-making to perform complex, multi-step tasks. Agents can interact with external APIs and services while maintaining context across interactions

Prompt injection is a security vulnerability where malicious users manipulate LLM inputs to override intended behavior, bypass safety measures, or extract sensitive information. Attackers embed instructions within data to make models ignore original prompts and follow malicious commands. Mitigation requires input sanitization, injection-resistant prompt design, and proper security boundaries.

Model weights and parameters are the learned values that define an LLM's behavior and knowledge. Parameters are the trainable variables adjusted during training, while weights represent their final values. Understanding parameter count helps gauge model capabilities - larger models typically have more parameters and better performance but require more computational resources.