Kafka-GCP Consideration

Kafka

• Functions similar to a log.
  • Messages are accessed by index/offset which increments after consumption.
  • This allows messages to be re-consumed by other consumers.
    • Non-zero chance that a consumer re-consumes the same message
  • As messages stay in the log, there exists a retention policy that’s configured by age or size of the log.
• Core concepts can be found here.
• Research on Kafka cluster/replication as it is currently a SPOF.
  • Have a Kafka instance per chain that stripes data of other chains into the other (n-k) instances of Kafka, similar to RAID 5 (illustrated in link), or
  • Round-robin replication (n replicates to n+1, wrap over).

GCP Pub/Sub

Understanding Pub/Sub

• Functions similar to queue.
  • Message pops from memory when subscriber acknowledges message.
  • It’s possible to replay messages.
• Core concepts can be found here.

Pub/Sub vs Pub/Sub Lite

	Pub/Sub	Pub/Sub Lite
Price [1]	~$2000/month	- Zonal: ~$169/month - Regional: ~$608/month
Reliability	Highest	- Zonal: Best effort - Regional: Similar SLA to normal Pub/Sub
Resources	Managed	Pre-provisioned
Data replication	- ≥ 2 zones - Best effort 3rd additional zone	- Zonal: No replication - Regional: Async backup to 2nd zone

Kafka vs Pub/Sub

	Kafka	Pub/Sub (Lite)
Deployment	- Cloud (Confluent) - On-premise (current)	- Cloud
Message delivery (default)	At least once	Exactly once
Message flow	Offset increments, message remains	Message deleted on acknowledgement
Message order	Within partitions / same key	Within topics

More information can be found here.

Dev

1. Migration from Kafka to Pub/Sub (link).
2. KafkaJS manual committing for exactly-once semantics (EOS) (link).
   1. Java example.
   2. librdkafka - C/C++ Kafka client to support EOS (link).
   3. Blizzard’s Node.js wrap on librdkafka (link).
3. Pub/Sub Client (Node.js) — not sure if it works with pub/sub lite.
4. Pub/Sub Lite library (Go/Java/Python)
5. Pub/Sub Lite API (REST)(RPC)