Identifying User Activity from Streams of Raw Events

Mark M Mims | @m_3

The SVDS Team

Key Takeaways

  • what's needed to understand user activity
  • pipeline architectures that support this analysis

Agenda

  • Ingest Events
  • Take Action
  • Recognize Activity

Background

(slides go down from here)

The Power of the Query Side

Infrastructure Aspirations

  • immutable
  • lazy
  • ...

Ingest Events

(slides go down from here)

Device Events

  • location
  • environment
  • telemetry
  • presence
  • status (disk is full)
  • ...

User Events

  • login
  • checkout
  • add friend
  • ...

Device Event


{
  "time_utc": "1457741907.959400112",
  "device_id": "c361-445b-b2f6-27f2eecfc217",
  "event_type": "environmental_info",
  "degrees_fahrenheit": "72",
  ...
}

User Event


{
  "time_utc": "1457741907.959400112",
  "user_id": "688b60d1-c361-445b-b2f6-27f2eecfc217",
  "event_type": "login",
  ...
}

Flat User Event


{
  "time_utc": "1457741907.959400112",
  "user_id": "688b60d1-c361-445b-b2f6-27f2eecfc217",
  "event_type": "button_pressed",
  "button_type": "one-click purchase",
  "item_sku": "1 23456 78999 9",
  "item_description": "Tony's Run-flat Tire",
  "item_unit_price": ...
  ...
}

Complex User Event


{
  "time_utc": "1457741907.959400112",
  "user_id": "688b60d1-c361-445b-b2f6-27f2eecfc217",
  "event_type": "button_pressed",
  "event_details": {
    "button_type": "one-click purchase",
    "puchased_items": [
      {
        "sku": "1 23456 78999 9",
        "description": "Tony's Run-flat Tire",
        "unit_price": ...
        ...
      },
    ],
  },
  ...
}

Take Action

(slides go down from here)

What kind of actions?

  • Notifications
  • Decorations
  • Routing / Gating
  • Counting
  • ...

When to take action?

  • "batch"
  • "real-time"

Recognize Activity

(slides go down from here)

What's activity?

Sequence of events

State

A Toaster

Netflix

Package State

Complex Package State

Best Practices

(slides go down from here)

The Power of the Query Side

Infrastructure Aspirations

  • immutable
  • lazy
  • atomic
    • simple
    • composable
    • testable

Automate All of the Things

Test All of the Things

Failure is a First Class Citizen

Wrap-up

  • Ingest Events
  • Take Action
  • Recognize Activity

Thank You!

Mark M Mims | @m_3