DOE_MSc

A/B Testing

Description

We want to check how different versions of our newsletter (which is send by e-mail) is changing the open rate. We have two versions of the newsletter.

We assign the newsletter recipients randomly into two groups. Each group receives a different version of the newsletter.

In the end we check which newsletter was opened more, thus this version wins and will be further used.

Advantages

• Planning and execution effort is low/simple.
• Easy to analyze the results of the test

Disadvantages

Depending on the prerequisites (number of variants, evaluation procedures, etc.) many observations are needed according to power analysis -> potential profit gets lost

Dynamic-algorithmic procedure

Description

We want to check how different versions of our newsletter (which is send by e-mail) is changing the open rate. We have two versions of the newsletter.

The experiment takes place over 4 days. We send out the newsletter every 6 hours. For the first batch, an equal count of recipients is assigned.

Depending on the success (the open rate), we change the allocation of recipients to the two different versions. The version which has the higher open rate receives more recipients. After 4 days we check which version has received more recipients. The version with the higher amount of recipients wins (= higher open rate)

Advantages

• Realtime data shows (potentially) quickly an outcome of the test
• The test can be adjusted 

Disadvantages

The dynamic process produces ‘only’ one best result.

- No further models can be estimated

- No dependence of the conversion rate on other factors can be investigated

Research design

A/B Testing cannot be used in all research questions, Ex. For a complete redesign of a website! (Quality aspects, survey, UX testing)

Sampling

Sampling bias / population is unknown, Example: Conducting an experiment on a webpage layout that includes only heavy users.

Sample size

Large samples Hypothesis test is significant, but information on effect sizes is missing.

Metrics

The measurement uses an unsuitable metric.

Dark patterns are patterns of persuasion and influence, examples:

• Misdirection: The design purposefully focuses your attention on one thing to distract your attention from something else.
• Price Comparison Prevention: The retailer makes it hard for you to compare the price of an item with another item, so you cannot make an informed decision.
• Privacy Zuckering: You are tricked into publicly sharing more information about yourself than you really intended to. Named after Facebook CEO Mark Zuckerberg

How to arrange Factors and to get information out of them. One or more independent variables (= factors) with several levels are investigated simultaneously.

Factors = independent variables

The research team has several factors with several levels in which they have an interest, and levels of all factors are varied simultaneously. As a result, all possible combination of factor levels can be studied. The factor levels are determined before the experiment.

-> Interaction can be tested!

---

---

---

---

The more factors and the more characteristics, the larger the number of groups. Provided that there are NO interactions, the experiment can be conducted with a reduced number of groups By using Latin squares or related design, the number of groups required can be reduced compared to a full factorial design

= Relationship between a complete design and a Latin square (standard form)

Level ^ factors -1

3^4 = 84 -> 3^3 = 27

a.  Each level of C is combined only once with each level of A and each level of B

-> Latin square is completely balanced in terms of the main effects

b.  Each level of C occurs partially with other combinations of A and B

-> Latin square is only partially balance with respect to first-order interaction

Volume (Datenmenge)

Describes the extreme amount of data. Increasing amounts of data must be stored and processed.

Important aspects for DoE

- Relation of the number of variables vs. sample size

- Relation of sample / administrative dataset vs. population

Velocity

(Geschwindigkeit)

Describes the increased speed with which the data must be produced, but also processed at the same time

Variety

(Vielfalt)

Describes the variety of different file structures: structured, semi-structured and unstructured.

Veracity

(Wahrhaftigkeit)

The uncertainty (veracity) of the data and data quality. In some cases, data from various sources does not arrive in the desired quality and can therefore not be used as intended or must be preprocessed at great expense

---

Made data

Are taken as part of a study, answers research question, data is generated by researcher

Found data

Collected for various reasons, serves documentary and administrative purposes, data is obtained administratively and technically

Grey area

Lies between: data from full surveys (=census), social media, representative surveys

Bias is the deviation (= Abweichung) between the population mean µ0 and the sample mean x. There are three elements determining the bias:

Calculating the deviation between the sample mean and the mean in the population produces:

• statistical paradises: Have large data quantities → Big data (absolute)
• paradoxes in big data: It is not about the absolute size, but the relation!
  • Often not the paradise we expected!

If the size n of the administrative dataset goes towards N, the bias becomes arbitrarily small. The absolute size n is meaningless without specifying N! Example: n = 2’000’000 “is big”, but the proportion is still small with f = n/N = 2’000’000 / 200’000’000 = 1%

Confounding (“Verwirrung”) means that a factor (confounder) that is not directly investigated is associated with both the independent variable and the dependent variable and accordingly causes the relationship between the two variables (Spurious Correlation = “falsche Korrelation”)

Example: The fewer storks nest in an area, the lower human birth rate in the area

• direct relationship is most unlikely
• explainable trough urbanization (confounder)
• Urbanization leads to a decline in the stork population because more land is being developed
• Urbanization also leads to a lower birth rate because women tend to be more educated

Solution: Randomization

These studies are the solution to the disadvantages of observational studies. Experimental studies are ones where researchers introduce an intervention and study the effects. The Randomized double blind placebo control study/trial (= RCT) is considered the “gold standard” for producing reliable evidence because little is left to chance.

Aim: Treatment/exposures is planned, and its effects are investigated

Eligible people are randomly assigned to one of two or more groups. This ensures the comparability of the groups at study begin. Aim of randomization:

• Elimination of selection bias: is an experimental error that occurs when the participant pool is not representative of the target population. (parents with higher incomes >> more informed >> get their child vaccinated >> proportion would be larger in the treatment group)
• Elimination of confounding: Confounding means that a factor (=confounder), that it not directly investigated, relates to both the independent and dependent variable. This confounder then causes the relationship between the two variables (= Spurious Correlation) and not the treatment itself.
• Ensuring the comparability of the groups at study initiation (baseline data)

Blinding is a suitable technique to avoid distortion (“Verzerrung”) due to knowledge. Blinding eliminates conscious and unconscious influences on the treatment results. Also it ensures the comparability of the groups during / in the course of the study trough the similar treatment of the two groups and the independent evaluation of the effect of the exposure. There are several stages of blinding (cumulative):

• Open: no blinding
• Single-blind: Tested person has no knowledge of group membership.
• Double-blind: The person carrying out the experiment has no knowledge of the group membership.
• Triple-blind: Person doing the analysis has no knowledge about the group membership.

In Short:

Effect and aim of blindness.

• Elimination of conscious and unconscious influences of the treatment result (Placebo)
• Ensuring comparability regarding treatment of groups during the study
• Ensuring comparability on the final evaluation of both groups

Descriptive study

Descriptive character, suitable for forming hypotheses but insufficient for testing them.

Cross sectional study

Sample with variables of interest are collected and described at a specific time “Quality of Life of master’s students.”

-> Survey is typical

Correlational study

Same as cross sectional, but variables are correlated:

“Relationship between the time that master’s students spend studying on their own and their general quality of life”

dentification and quantification of effects of relationship, not fully suitable for hypothesis testing.

Case control study                                                     

Examines how cases and controls differ from having been exposed previously Looking back at the exposition (retrospective)

Example with Pädos (Vergangenheit wird angeschaut, wieso wurden sie so?)

Cohort study

Cohort = group of people with comparable initial conditions

One group is exposed to an influencing factor afterwards the groups are compared

Randomized controlled (RCT)

Treatment and control are determined by a random process (randomization) before the intervention.

uitable for hypothesis testing.

Non-randomized controlled

Treatment and control are not determined by a random process (randomization) before the intervention. For example, control groups involve data obtained from previous studies («historical control») or from randomly allocated groups.

Input

Trial objects, tests objects, test persons, materials

Process

Process in which controllable and non-controllable factors influence the input

Controllable factors

Influencing factors whose strength can be adjusted within defined limits.

• Independent variable (IV)

Non-controllable factors

Influencing factors whose strength cannot be determined,

• but measured:
  • body weight, stress level, etc.
• or not measured:
  • random fluctuation in output, random fluctuation in measurement instruments

• Nuisance variables (= Störvariablen)

Output

Input changed by the process, result of the test/experiment.

• Dependent variable (DV)

Impact of factors on the variation of the output (focus of the study)

Ex. Body weight changes systematically when height changes:

Variation of the output, caused by nuisance factors (not in the focus of the study, for example gender)

Ex. Boys and girls differ in the level of body weight, that’s why we can divide the sample into two groups

Variation caused by measurement errors or random processes.

The Variance of the dependent variable (primary variance) should be attributed to the systematic variation of the independent variable (IV). The secondary variance should be controlled, and the error variance minimized.