Building ETL Pipelines with Apache Beam: Go vs Java vs Python

Dean Jain

Senior Staff Software Engineer · Enterprise AI, Data & Cloud Architect

October 7, 2023 · 6 min read

Apache BeamDataflowETLData Engineering

---
config:
  theme: dark
  fontSize: 17
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
---
flowchart LR
    PIPE["🧩 One pipeline model<br/>PCollection + DoFns"]:::gov --> SDK["🛠️ SDK<br/>Go · Java · Python"]:::gate
    SDK --> RUN["🏃 Any runner<br/>Dataflow · Direct · Flink · Spark"]:::good
    classDef gov fill:#E0D6F5,stroke:#9B7EDE,stroke-width:2px,color:#0F172A
    classDef gate fill:#D7C3F2,stroke:#8E5BD0,stroke-width:2px,color:#0F172A
    classDef good fill:#BFEFC8,stroke:#3FA34D,stroke-width:2px,color:#0F172A

Figure 1: Beam's promise. Define the pipeline once, write it in your language, run it on any runner.

Apache Beam’s pitch is unusual. One unified model for both batch and stream processing. Written in the language you choose, executed on the runner you choose: Dataflow, Flink, Spark, or local Direct. To put that to the test, here’s the same real ETL pipeline, paginating data out of Elasticsearch and bulk-writing it into Firestore, implemented in Go, Java, and Python.

The valuable lesson isn’t the code. It’s seeing what stays identical, which is the Beam concepts, and what changes, which is SDK maturity and idiom. That is how you pick the right SDK for your team.

TL;DR

Beam separates the what from the where. You define a pipeline of PCollections and DoFns; the runner decides how and where it executes. Same code on Direct locally or Dataflow in the cloud.
The DoFn lifecycle is universal across languages: Setup → StartBundle → Process → FinishBundle → Teardown. Manage clients accordingly.
The ETL is the same shape in all three. Options-driven config, a DoFn to read ES with search_after, a DoFn to write Firestore with BulkWriter, and tagged outputs for errors.
The SDKs differ in maturity and idiom. Java is the most feature-rich. Python is the most popular for data work. Go is the leanest and youngest, with some pattern gaps.
Same pipeline, three flavors so the choice is about your team’s language and ecosystem, not capability.

1. The Beam model: one pipeline, any runner

The idea that makes Beam portable is the separation of the pipeline definition from its execution. You describe data as immutable PCollections flowing through transforms (mostly ParDo applying your DoFns), and that description is runner-agnostic. The runner takes that same graph and runs it: DirectRunner for local testing, Dataflow for managed cloud execution at scale, or Flink and Spark. It handles parallelism, sharding and retries for you, and on Dataflow it autoscales too. DirectRunner is the exception: it optimizes for correctness over performance and is not meant for production.

For our ETL, the pipeline shape is identical regardless of language:

Pipeline options drive configuration (ES host, index, query, Firestore project, target collection, partition count) no hardcoding.
A DoFn gets the initial _count and starts a tracking log.
A DoFn reads Elasticsearch in pages via search_after.
A DoFn writes to Firestore with BulkWriter.
Tagged side outputs route errors to a separate PCollection.

That’s the payoff of the model: you reason about the dataflow once, and Dataflow turns it into a parallel, autoscaling job. The language you write it in changes the syntax and some ergonomics but not this structure.

2. The `DoFn` lifecycle, the same in every SDK

One concept matters more than the rest, and it’s the same across Go, Java, and Python. It’s the DoFn lifecycle. Beam calls specific hooks at specific times, and you place resource management accordingly:

---
config:
  theme: dark
  fontSize: 17
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
---
flowchart LR
    SU["⚙️ Setup<br/>create clients (once)"]:::gov --> SB["📦 StartBundle<br/>per-bundle setup (BulkWriter)"]:::server
    SB --> PR["🔁 Process<br/>handle each element"]:::obs
    PR --> FB["✅ FinishBundle<br/>flush the batch"]:::gate
    FB --> TD["🧹 Teardown<br/>close clients"]:::good
    classDef gov fill:#E0D6F5,stroke:#9B7EDE,stroke-width:2px,color:#0F172A
    classDef server fill:#A8E6D0,stroke:#2FA37C,stroke-width:2px,color:#0F172A
    classDef obs fill:#AED6F1,stroke:#2E86C1,stroke-width:2px,color:#0F172A
    classDef gate fill:#D7C3F2,stroke:#8E5BD0,stroke-width:2px,color:#0F172A
    classDef good fill:#BFEFC8,stroke:#3FA34D,stroke-width:2px,color:#0F172A

Figure 2: The DoFn lifecycle. Create expensive clients in Setup, manage per-batch resources in Start/FinishBundle, clean up in Teardown.

The rule that matters: match each resource to the right hook.

Setup and Teardown run once per DoFn instance. Create expensive, long-lived clients in Setup: the Firestore client, or an HTTP client whose connection pool you want reused. Close them in Teardown, but don’t depend on it alone, because Beam runs Teardown on a best-effort basis and doesn’t guarantee it.
StartBundle and FinishBundle run per bundle, which is Beam’s unit of work. This is exactly where a Firestore BulkWriter belongs: create it in StartBundle, flush it in FinishBundle. Aligning the BulkWriter’s lifecycle with the bundle is what makes batched writes efficient and correct.
Process runs per element keep it lean; do the per-record work and emit results (or errors via tagged outputs).

Getting this wrong is the most common Beam ETL bug e.g. creating a Firestore client per element (catastrophic) or never flushing the BulkWriter (lost writes). The lifecycle is identical in all three SDKs, so once you internalize it, you can read and write Beam pipelines in any of them.

A shared gotcha worth flagging: search_after pagination is inherently sequential, which clashes with Beam’s parallel model. All three implementations hit this. A DoFn emits the next page’s parameters. Looping that back into the pipeline graph needs an advanced pattern: Splittable DoFn, stateful DoFn, or the Elasticsearch slice API for parallel scrolling. The pragmatic answer in every language: fetch sequentially, but parallelize the processing and writing of each fetched batch.

3. Go vs Java vs Python: same pipeline, three flavors

The structure is identical; the feel differs. Where each SDK lands:

---
config:
  theme: dark
  fontSize: 17
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
---
flowchart TD
    P["🧩 Same ES→Firestore pipeline"]:::gov
    P --> J["☕ Java<br/>most mature, feature-rich"]:::server
    P --> PY["🐍 Python<br/>most popular for data/ML"]:::obs
    P --> GO["🦫 Go<br/>leanest, youngest"]:::gate
    classDef gov fill:#E0D6F5,stroke:#9B7EDE,stroke-width:2px,color:#0F172A
    classDef server fill:#A8E6D0,stroke:#2FA37C,stroke-width:2px,color:#0F172A
    classDef obs fill:#AED6F1,stroke:#2E86C1,stroke-width:2px,color:#0F172A
    classDef gate fill:#D7C3F2,stroke:#8E5BD0,stroke-width:2px,color:#0F172A

Figure 3: The same pipeline across three SDKs. Choose by team and ecosystem, since all three can express it.

	Java	Python	Go
Maturity	Most mature, richest API & connectors	Strong, widely used	Youngest; some patterns missing
Ecosystem fit	Enterprise JVM shops	Data science / ML	Go-native backends
ES client	`CloseableHttpClient` + pooling	`requests.Session`	`net/http` (pools internally)
Firestore	`BulkWriter`	client `BulkWriter`	`cloud.google.com/go/firestore` BulkWriter
DoFn style	Classes	Classes/functions	Structs (must `beam.RegisterType`)
Errors	Tagged outputs	Tagged outputs	Tagged side outputs

Java is the most mature SDK the broadest set of built-in I/O connectors and the most battle-tested patterns. The natural choice for JVM/enterprise teams.
Python is the most popular for data and ML work, with clean PipelineOptions and easy integration with the analytics ecosystem. Some advanced parallel-extraction patterns, Splittable DoFns among them, are more involved.
Go is the leanest and youngest. DoFns are structs, and you must beam.RegisterType them so Beam can serialize them. Config uses the standard flag package, and clients are idiomatic Go. The trade-off is its relative youth some patterns (like the pagination loop) need more manual effort, and the connector set is smaller.

The takeaway is simple. All three express the same pipeline. So the SDK choice is about your team’s language and ecosystem, not about what Beam can do. Pick the language your team already operates well in.

Beam’s real value is decoupling what your pipeline does from where and how it runs. Write the dataflow once, let Dataflow scale it, in the language your team already knows. Internalize the universal parts (the pipeline/PCollection/DoFn model and the DoFn lifecycle) and you can read or build a pipeline in any SDK. The Go-vs-Java-vs-Python question then stops being about capability and becomes the easy kind of decision: which ecosystem fits your team.

TL;DR

1. The Beam model: one pipeline, any runner

2. The DoFn lifecycle, the same in every SDK

3. Go vs Java vs Python: same pipeline, three flavors

Further reading

2. The `DoFn` lifecycle, the same in every SDK