FC

Creates a pseudorandomized but deterministic distribution of replicas across machines

Pro

• Can be used for good placement in geo-distributed deployments

Contra

• Poorly equipped to deal with end-user mobility and resulting access pattern across system nodes

Example

• CRUSH

  • Computes a pseudorandom data placement distribution based on a hierarchical description of the target cluster

The four strategies are not a natural fit for the fog

Combination of different strategies is the way to go

Examples

• PNUTS

  • Global mapping for replica placement within a region and full replication across regions

• DynamoDB

  • Offers global mapping for cross-region replication

• FARSITE

  • Combines global mapping with scattering

• IPFS

  • Bit-Torrent inspired protocol for data exchange and a hashing algorithm to determine storage locations

IPFS is a peer to peer distributed file system

Object are content-addressable

Merkle DAG

Advantages

• Content-based Addressing

• Tamper-proof

  • Content is verified with a checksum

• No duplication

  • Object with the same content have the same ID

Technologies

• DHT

• Block Exchange - Bit Torrent

• Version Control SyStems - Git

• Self-Certifying Filesystem

Not fog-ready: because of the slow DHT

• Use a scale-out NAS (RozoFS) to enable site reads without using the DHT

Data-centric abstraction focused on the distribution, preservation, and protection of information

Builds upon append-only, single-writer logs

• Lightweight and durable

• Multiple simultaneous reads

• No fixed location, migrated as necessary

• Compositions are achieved by subscriptions

Location-independent routing

• Large 256-bit address space

• Packages are routed through an overlay network that uses a DHT

  • Enables flexible placement, controllable replication and simple migration of logs

• GDP places logs within the infrastructure and advertises the location to the underlying routing layer

• Placement and replication of logs can be optimized for latency, QoS. privacy and durability

• Logs themselves are split into chunks whose placement can be optimized for durability and performance

Application controlled replica placement

Key abstractions

• Nodes

  • Group of one or more machines within one geographical site

  • Including a hosed or embedded storage system

  • Nodes only interact with other nodes as a whole (not with individual machines)

  • Coordination within nodes is done through the storage system

• Keygroups

  • Group of data items that are replicated together

  • Own ACL

  • Applications declarative specify the set of key group members which controls data distribution

• Keygroup members

  • One or both roles:

  • Replica nodes

    • Store a data replica, serve client requests, and manage keygroup configuration

  • Trigger nodes

    • Receive all updates as a stream of events and may trigger external systems via an event-based interface

  • Applications can specify a TTL for data retention on replica nodes

Higher utilization of sacred edge resources

Stateless functions can be moved as needed

Event-driven is a good fit for many edge/fog applications

Flexibility

OpenWhisk to heavy for Fog

Replaced the heaviest components

If sully compatible with OpenWhisk and part of the OpenWhisk releases

Cons: Still a lot of unnecessary code that was written for cloud-based deployments

Goals: lightweight, extensible, http or http compatible

Key mechanisms

• Remove as many components as possible

• CoAP as application protocol

• Parallel execution of request within a container

  • On container per client or per function

Experiments

• Compare tinyFaaS to: native node.js, Lean OpenWhisk, Kubeless

• Infrastructure: Raspberry Pi 3 B+

• Measure latency at different load levels with hard SLA

• Results

  • Native node.js: very low overhead of tinyFaaS

  • Lean OW: does not work on a RaspberryPi

  • Kubeless: comparable at very low load, but does not scale

Suffice for the edge?

• Pro

  • Designed for small nodes

  • Much more efficient than alternative solutions

• Contra

  • No support for cluster-based or on-device deployment

Targeted at extremely resource-constrained devices

Subset of standard libraries

Implements AWS Lambda API

Builds on CSPOT

Lightweight python VM for bytecodes (IoTPy)

Remove compilation (cloud/edge): code is never delivered to device

Life cycle of function

• Deploy to NanoLambda Cloud/Edge service

  • Stores code

  • Compiles and caches compact bytecode representation on-demand

• IoT device requests bytecode from NanoLambda service

Suffice for the edge?

• Pro

  • Designed for small edge nodes and on-device

  • Cloud compatibility

• Contra

  • No support for cloud-based clusters

Testing

• Testing stage

• Method/function level

• Focus on functional behavior

• + Monitoring: Live testing

• + Benchmarking: Performance test / Microbenchmarks

Benchmarking

• Testing stage

• System level

• Stress test

• Focus on QoS

• + Monitoring: Service / API benchmarking

Monitoring

• Production stage

• System level

• Passive observation

• Focus on QoS

Unit testing

Integration & Live Testing

• Canary Testing

• Dark Launches

• A/B Testing

Cloud integration tests

• Moch services, data, devices

• Evaluate corner-cases which usually should not exist in production

Fog Integration tests

• Much more difficult because of physical infrastructure

• (partial) solution: virtualize & emulate fog environment in the cloud

Test new software version in production

Monitor what happens

• While rolling out an update gradually

• While directing part of the traffic to old and/or new version

Example

• Blue/Green Deployments

  • Deploy new version to blue environment

  • Smoke tests against the blue system

  • Switch traffic from green to blue

  • Switch back to green on errors

• Canary releasing

  • Rollout of a new version only to a subset of production servers

  • Easy to revert

  • Use it for A/B testing

  • Check capacity requirements by incrementally increasing the load

• Dark/Shadow Launches

  • Functionality is deployed in a production env without being visible or activated

  • Production traffic is duplicated and routed to the shadow version as well

  • Observing the shadow version without imparting the user

Cloud part: ok

Difficult on edge devices which

• May not have the capacity to run two versions in parallel

• May have safety requirements which make canary releases impossible

Find a separate solution for the edge part

• Mock edge devices in the cloud

• Have a physical testbed

Deploying cloud applications

• Changes are pushed to devices via IaC

• New virtual devices are created, configured and deployed with new version

  • Old instances are disconnected/terminated

Deploying fog applications

• Edge devices often need to be physically connected at least once for deploying the first version

• Use an app store-like approach

  • Update is sent to central software repository

  • Deployed application frequently checks for updates and self-updates if necessary => pull approach

• Plan with incompatibilities and different version on devices

• Used versioned interfaces

Benchmarking is a way to systematically study the quality of cloud services based on experiments

Benchmarking tool creates an artificial load on the SUT, while carefully tracking detailed quality metrics

Relevance

• Benchmark the important parts

• Mimic real-world use

Repeatability

• Maximize determinism in the benchmark

Fairness

• Treat all SUTs the same

Portability

• Avoid assumptions about the SUT

• Make the benchmark broadly applicable

Understandability

• Have an intuitive benchmark specification

Geo-distribution of experiments

Deployment of benchmarking clients for edge-based SUTs

Distributed measurements of QoS

• E2E latency in an IoT data processing pipeline

Multi-workload scenarios

• Event-driven at the edge

• OLAP and OLTP in the cloud

Complex analysis and results

Correctness

• Assert adherence of implementation to specification

Distribution

• Build the benchmarking tool for distributed deployments

• Keep coordination pre-benchmark run

• Consider clock synchronization

Fine-grained logging

• Never discard information if not absolutely necessary

Reproducibility

• Use repeatable benchmarks

• Repeat often

• Run sufficiently long

• Document setting

Portability

• Use adapter design

• Consider extensibility and evolvement

• Avoid assumptions on the SUT

Ease of use

• Document everything

• Provide instructions

• Release code

Microservice-based design

Infrastructure automation

Fault-tolerance through replication

Cluster-based deployment only in a few datacenters

• Fog: single-node to cluster sized deployments on millions of sites

Geo-awareness in the cloud

• Limited to large regions

• High latency if the closest data center is quite far

• Introduction Fog nodes

  • Fast connection to nearby fog nodes but limited bandwidth to cloud

  • Access points of mobile devices must be adapted based on their location

Geo-awareness

• Infrastructure needs to expose location and network topology explicit

Fault tolerance in cloud applications

• Redundant servers

• Retry-on-error principle (with other service instances)

• Monitor services and their workload, auto-scaling

• Chaos-Monkey randomly shuts down services to check if the system adapts and catches outage

Fault tolerance in fog applications

• The prevalence of faults depends on the number of nodes

  • Systems and/or their components fail continuously

  • Connection infrastructure fails or operates with reduces quality

    • Power outage

    • Some devices transmit data under certain conditions (sunlight)

    • Eventual consistency problems may result in stale datasets

  • Buffer messages until its receiver is available again

  • Expect data staleness and ordering issues

  • Cache data aggressively

  • Compress data items as much as possible on unreliable connections

  • Plan with incompatibility, constantly monitor software versions on devices

  • Design for loose coupling

Must be aware of its deployment location

Needs to handle client movement (handover to other edge devices)

Must be prepared to move components elsewhere (stateless application logic)

Must move data when necessary

May not rely on the availability of remote components

Motivation

• Application can be deployed in different ways

• Various hardware options exist on the edge

• How can we compare them?

Deployment options

• Three deployment modes

  • Cloud only

  • Edge only

  • Cloud-Edge (Fog)

• Docker as deployment vehicle

Approach

• Use a set of representative benchmark applications

• Measure E2E performance as well low-level metrics

• 6 applications

  • Latency critical

  • Bandwidth intensive

  • Location aware

  • Compute intensive

Benchmarking fog-based FaaS platforms

Federated deployments

• Different cloud provider

Workloads

• E-commerce application

• IoT application

How evaluate a fog application?

• Without testing infrastructure

  • Guesses, small local testbeds, and simulation

• Operate additional edge machines

  • Expensive, must be at same sites as production machines

• Idea: Us an emulated fog infrastructure testbed that is set up in the cloud

  • Size/Power of Vms: Cloud instance types and Docker resource limits

  • Network characteristics: tc, iptables, etc.

• MockFog

Three modules

• Infrastructure emulation

• Application management

• Experiment orchestration

  • Compromises a finite set of states

  • Failure testing

Node Manager and Node Agents