Log Analytics Pipelines as-a-Service

Confluent Kafka and Elasticsearch PersistentVolumes orchestrated by Portworx while also using S3 buckets for long-term shared storage. The PortWorx storage can be backed by local drives, FlashArray volumes, or FlashBlade NFS.

How Shared Storage Simplifies aaS Log Pipelines

Object Storage

PersistentVolume Dynamic Provisioning

Log Pipeline Components

Prerequisites

FlashBlade Configuration

Flog: Synthetic Log Generator

> docker run -it --rm mingrammer/flog

137.97.114.3 — — [27/Aug/2020:19:50:11 +0000] “HEAD /brand HTTP/1.1” 416 16820
252.219.8.157 — — [27/Aug/2020:19:50:11 +0000] “PUT /maximize/synergize HTTP/1.0” 501 4208

Confluent Kafka

FileBeats

Elasticsearch

Example values.yaml File

flashblade:
datavip: "10.62.64.200"
mgmtvip: "10.62.64.20"
token: "T-XXXXXX-YYYYY-ZZZZ-QQQQQ-813e5a9c1222"
zookeeper:
storageclass: "px-nvme"
kafka:
cpVersion: 6.1.1
storageclass: "px-nvme"
nodecount: 4
elasticsearch:
nodecount: 6
version: 7.12.1
storageclass: "pure-file"
beats:
nodecount: 12
flog:
nodecount: 1

How to Adapt For Your Use Case

Storage Usage Visualized

Write (orange) and Read (blue) performance for Elasticsearch ingest
Write spikes to S3 as indices are moved to the Frozen Tier

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store