Pipes and Filters Pattern: Data Processing Pipelines

TL;DR

The Pipes and Filters pattern decomposes complex data processing into a chain of independent, reusable filters connected by pipes. Each filter performs a single transformation, passing results through pipes to the next stage. This enables parallel execution, independent scaling, and easy reconfiguration of processing logic.

Cheat Sheet:

• Pattern Type: Architectural pattern for data processing
• Key Benefit: Decomposition enables independent scaling and reuse
• Best For: ETL pipelines, media processing, log aggregation
• Watch Out: Overhead from data serialization between filters
• Interview Tip: Explain how you’d handle backpressure and filter failures

The Problem It Solves

Complex data processing systems often become monolithic nightmares. Imagine building a video transcoding service where you need to decode input, resize frames, apply watermarks, encode to multiple formats, and upload results. The naive approach creates a single massive function that does everything. When you need to add HDR support, you’re modifying a 3000-line method. When encoding becomes the bottleneck, you can’t scale just that step—you must scale the entire monolith. When the watermarking logic crashes, it takes down the whole pipeline.

This tight coupling creates three critical problems. First, you can’t reuse processing logic across different workflows—the resize logic is buried in the video pipeline, unavailable for image processing. Second, you can’t independently scale or deploy components—if encoding needs 10x more resources than decoding, tough luck. Third, testing becomes a nightmare because you can’t validate individual transformations in isolation. Netflix faced exactly this when building their encoding pipeline: a monolithic transcoder that couldn’t adapt to new codecs or scale components independently. The solution required rethinking how data flows through processing stages.

Solution Overview

The Pipes and Filters pattern breaks complex processing into a chain of independent filters connected by pipes. Each filter is a self-contained component that receives input, performs one transformation, and produces output. Pipes are the conduits that move data between filters, handling buffering and potentially format conversion. The key insight is that filters know nothing about their neighbors—they just consume from an input pipe and produce to an output pipe.

Think of it like Unix command-line pipes: cat access.log | grep ERROR | awk '{print $1}' | sort | uniq -c. Each command is a filter doing one thing well. The pipe operator connects them. You can rearrange, add, or remove filters without touching their internals. The same principle applies at system scale. Netflix’s encoding pipeline has filters for decoding, resizing, encoding, and packaging. Each filter can be written in different languages, scaled independently, and deployed without affecting others. When they added 4K support, they only modified the resize and encode filters. The decode and package filters remained unchanged.

Basic Pipes and Filters Architecture

graph LR
    Input["Raw Data<br/><i>Input Source</i>"]
    
    subgraph Pipeline
        F1["Filter 1<br/><i>Parse</i>"]
        P1["Pipe<br/><i>Queue/Stream</i>"]
        F2["Filter 2<br/><i>Transform</i>"]
        P2["Pipe<br/><i>Queue/Stream</i>"]
        F3["Filter 3<br/><i>Aggregate</i>"]
    end
    
    Output["Processed Data<br/><i>Output Sink</i>"]
    
    Input --"1. Raw input"--> F1
    F1 --"2. Parsed data"--> P1
    P1 --"3. Buffered data"--> F2
    F2 --"4. Transformed data"--> P2
    P2 --"5. Buffered data"--> F3
    F3 --"6. Final output"--> Output

The core pattern showing independent filters connected by pipes. Each filter performs one transformation and passes results through a pipe to the next stage. Filters are unaware of their neighbors, enabling independent development and scaling.

How It Works

Step 1: Define Filter Interfaces

Every filter implements a standard interface with process(input) -> output. For a log processing pipeline, you might have: ParseFilter (raw text → structured JSON), EnrichFilter (add geo-location from IP), FilterByLevel (keep only ERROR/WARN), and AggregateFilter (count by error type). Each filter is a separate class or service that knows only its input and output contracts.

Step 2: Connect Filters with Pipes

Pipes handle data movement between filters. In-memory implementations use queues or channels. Distributed systems use message brokers like Kafka or RabbitMQ. The pipe abstracts the transport mechanism—filters don’t know if they’re reading from memory or a message queue. For our log pipeline: RawLogs → [ParseFilter] → StructuredLogs → [EnrichFilter] → EnrichedLogs → [FilterByLevel] → ErrorLogs → [AggregateFilter] → Metrics.

Step 3: Implement Backpressure

When a slow filter can’t keep up, pipes must handle backpressure. If the AggregateFilter processes 1000 logs/sec but EnrichFilter produces 5000/sec, the pipe between them fills up. Solutions include: blocking the producer (EnrichFilter waits), buffering with bounded queues (drop oldest or reject new), or dynamic scaling (spin up more AggregateFilter instances). Kafka handles this naturally with consumer lag metrics.

Step 4: Handle Errors Gracefully

Filters fail. The EnrichFilter might timeout calling the geo-location API. Options include: retry with exponential backoff, send to a dead-letter queue for manual review, or skip and continue (with logging). Netflix’s pipeline uses a hybrid: transient errors retry 3 times, permanent errors (corrupt video files) go to DLQ, and the pipeline continues processing other videos.

Step 5: Enable Dynamic Reconfiguration

The power of this pattern emerges when you can swap filters at runtime. Facebook’s image processing pipeline lets product teams register custom filters. When Instagram wants to add a new filter effect, they deploy a new filter service and update the pipeline configuration—no changes to existing filters. The pipeline becomes a composable data processing framework.

Log Processing Pipeline with Backpressure

graph LR
    subgraph Input Stage
        Logs["Raw Logs<br/><i>5000/sec</i>"]
    end
    
    subgraph Parse Stage
        Parse["ParseFilter<br/><i>5000/sec capacity</i>"]
        Q1["Queue<br/><i>Max: 10K msgs</i>"]
    end
    
    subgraph Enrich Stage
        Enrich["EnrichFilter<br/><i>3000/sec capacity</i>"]
        Q2["Queue<br/><i>Max: 10K msgs</i>"]
    end
    
    subgraph Filter Stage
        FilterLevel["FilterByLevel<br/><i>2000/sec capacity</i>"]
        Q3["Queue<br/><i>Max: 5K msgs</i>"]
    end
    
    subgraph Aggregate Stage
        Agg["AggregateFilter<br/><i>1000/sec capacity</i>"]
    end
    
    DLQ["Dead Letter Queue<br/><i>Failed records</i>"]
    Metrics["Metrics Store"]
    
    Logs --"1. Stream"--> Parse
    Parse --"2. JSON"--> Q1
    Q1 --"3. Consume"--> Enrich
    Enrich --"4. Enriched"--> Q2
    Q2 --"5. Consume"--> FilterLevel
    FilterLevel --"6. Errors only"--> Q3
    Q3 --"7. Consume"--> Agg
    Agg --"8. Counts"--> Metrics
    
    Enrich -."Retry 3x".-> Enrich
    Enrich -."Permanent failure".-> DLQ
    Q2 -."Backpressure: Queue full".-> Enrich

A complete log processing pipeline showing throughput mismatches and backpressure handling. When EnrichFilter (3000/sec) can’t keep up with ParseFilter (5000/sec), Queue Q1 fills up and blocks the producer. Failed enrichments retry 3 times before moving to the dead-letter queue.

Error Handling and Recovery Strategies

stateDiagram-v2
    [*] --> Processing: Record arrives
    
    Processing --> Success: Filter completes
    Processing --> TransientError: Timeout/Network error
    Processing --> PermanentError: Invalid data/Corrupt file
    
    TransientError --> Retry1: Attempt 1
    Retry1 --> Success: Succeeds
    Retry1 --> Retry2: Fails (wait 1s)
    Retry2 --> Success: Succeeds
    Retry2 --> Retry3: Fails (wait 2s)
    Retry3 --> Success: Succeeds
    Retry3 --> DeadLetterQueue: Fails (wait 4s)
    
    PermanentError --> DeadLetterQueue: No retry
    
    Success --> [*]: Continue pipeline
    DeadLetterQueue --> ManualReview: Alert ops team
    ManualReview --> [*]: Resolved
    
    note right of TransientError
        Examples:
        - API timeout
        - Network blip
        - Rate limit
    end note
    
    note right of PermanentError
        Examples:
        - Corrupt video
        - Invalid JSON
        - Missing required field
    end note

Error handling state machine for filters. Transient errors (timeouts, network issues) retry with exponential backoff up to 3 attempts. Permanent errors (corrupt data, validation failures) skip retries and go directly to the dead-letter queue for manual review, allowing the pipeline to continue processing other records.

Variants

Linear Pipeline

Filters arranged in a strict sequence where each filter has exactly one input and one output. This is the simplest variant, used for straightforward transformations like ETL jobs. When to use: Processing has a clear sequential flow with no branching. Pros: Easy to reason about, simple error handling. Cons: Can’t parallelize independent operations, single failure point.

Tee Pipeline

A filter’s output splits to multiple downstream filters. After parsing logs, one pipe goes to real-time alerting, another to long-term storage. Twitter’s firehose uses this: incoming tweets split to multiple consumer pipelines (search indexing, timeline fanout, analytics). When to use: Multiple independent consumers need the same data. Pros: Enables parallel processing, consumers don’t block each other. Cons: Coordination complexity if consumers need to synchronize.

Conditional Pipeline

Filters route data based on content. After parsing, valid records go to processing, invalid to error handling. Stripe’s payment pipeline routes transactions differently based on risk score: low-risk gets fast-tracked, high-risk goes through additional verification filters. When to use: Different data requires different processing paths. Pros: Efficient resource usage, specialized handling. Cons: Routing logic can become complex, harder to trace data flow.

Feedback Loop Pipeline

Output from a late-stage filter feeds back to an earlier stage. Machine learning training pipelines use this: predictions feed back to improve the model. When to use: Iterative refinement or adaptive processing. Pros: Enables self-improving systems. Cons: Risk of infinite loops, harder to test and debug.

Pipeline Variants: Linear, Tee, and Conditional

graph TB
    subgraph Linear Pipeline
        L1["Input"] --> L2["Filter A"]
        L2 --> L3["Filter B"]
        L3 --> L4["Filter C"]
        L4 --> L5["Output"]
    end
    
    subgraph Tee Pipeline
        T1["Input"] --> T2["Parse"]
        T2 --> T3["Consumer 1<br/><i>Real-time alerts</i>"]
        T2 --> T4["Consumer 2<br/><i>Storage</i>"]
        T2 --> T5["Consumer 3<br/><i>Analytics</i>"]
    end
    
    subgraph Conditional Pipeline
        C1["Input"] --> C2["Validate"]
        C2 --"Valid"--> C3["Process"]
        C2 --"Invalid"--> C4["Error Handler"]
        C3 --> C5["Low Risk"]
        C3 --> C6["High Risk<br/><i>Extra verification</i>"]
        C5 --> C7["Output"]
        C6 --> C8["Manual Review"]
        C8 --> C7
    end

Three common pipeline variants. Linear pipelines process sequentially. Tee pipelines split output to multiple independent consumers (Twitter’s firehose). Conditional pipelines route data based on content (Stripe’s risk-based payment routing).

Trade-offs

Modularity vs Overhead

Each filter boundary adds serialization/deserialization cost. A monolithic function processing 1M records might take 10 seconds. The same logic split into 5 filters with JSON serialization between them might take 15 seconds due to marshaling overhead. However, you gain independent deployability and reusability. Decision criteria: Use pipes and filters when flexibility and maintainability outweigh the 20-50% performance overhead. For ultra-low-latency paths (< 1ms), consider monolithic processing.

Independent Scaling vs Coordination Complexity

You can scale each filter independently—run 10 instances of the slow encoder, 2 of the fast decoder. But now you need service discovery, load balancing, and monitoring for each filter. A monolith needs one deployment, one monitoring dashboard. Decision criteria: Use independent scaling when different stages have vastly different resource needs (CPU-heavy encoding vs I/O-heavy uploading) or when traffic patterns vary (more reads than writes).

Reusability vs Coupling

Filters are reusable across pipelines only if they’re truly independent. But real systems have shared state—caches, databases, configuration. The EnrichFilter might cache geo-location lookups. Do other pipelines share that cache? Now you’ve introduced coupling. Decision criteria: Design filters as pure functions when possible. For stateful filters, use external state stores (Redis, databases) that multiple pipelines can safely share.

Flexibility vs Complexity

Adding a new processing step means deploying one new filter, not modifying existing code. But debugging becomes harder—a bug might be in any of 10 filters. Tracing a single record through the pipeline requires distributed tracing. Decision criteria: Accept the complexity when processing logic changes frequently or when multiple teams own different stages. Stick with monoliths for stable, rarely-changing workflows.

When to Use (and When Not To)

Use Pipes and Filters When:

Your processing has clear sequential stages with different resource requirements. Video transcoding is the poster child: decode (CPU), resize (GPU), encode (CPU), upload (network). Each stage scales independently. Use it when different teams own different processing stages—the ML team owns the recommendation filter, the content team owns the moderation filter. Use it when you need to A/B test processing logic—run 90% of traffic through the old filter, 10% through the new one. Use it for batch and stream processing that shares logic—the same filters work on Kafka streams and S3 files.

Avoid When:

Processing requires tight coupling between stages. If step 2 needs to look back at step 1’s intermediate results, filters become awkward. Avoid for ultra-low-latency requirements (< 5ms) where serialization overhead matters. Avoid when processing is trivial—three lines of code don’t need five filters. Avoid when you can’t tolerate eventual consistency—if all stages must succeed or rollback atomically, use a transaction, not a pipeline.

Anti-Patterns:

Creating filters that are too granular (one filter per line of code) adds overhead without benefit. Creating filters that do too much (a “processing” filter that does 10 things) defeats the purpose. Sharing mutable state between filters through side channels instead of pipes creates hidden dependencies. Ignoring backpressure until the system crashes under load.

Real-World Examples

Netflix: Video Encoding Pipeline

Netflix processes millions of video files through a pipes and filters architecture. The pipeline starts with a validation filter (check file integrity), followed by decode (extract raw frames), analyze (detect scene changes, black frames), encode (transcode to multiple bitrates and codecs), package (create HLS/DASH manifests), and upload (to CDN). Each filter is a separate microservice that can be written in the optimal language—decode in C++ for performance, analyze in Python for ML libraries. When they added AV1 codec support, only the encode filter changed. The interesting detail: they use a “tee” variant where the analyze filter’s output goes to both the encode filter and a separate quality metrics pipeline, enabling parallel processing without blocking the main encoding flow.

Facebook: Image Processing Pipeline

Facebook’s photo upload pipeline uses pipes and filters to handle billions of images. The pipeline includes: upload validation, virus scanning, EXIF stripping (remove metadata), resize (generate thumbnails and multiple resolutions), face detection, content moderation, and storage. Each filter is independently scalable—face detection runs on GPU instances, resize on CPU instances. The clever part: they use conditional routing where images flagged by content moderation go through a human review filter before storage, while clean images skip directly to storage. This reduces review costs by 90% while maintaining safety. Filters are implemented as stateless functions that can run on any server, enabling massive horizontal scaling during peak upload times.

Netflix Video Encoding Pipeline

graph LR
    Input["Video Upload<br/><i>S3</i>"]
    
    subgraph Validation Stage
        V["Validate<br/><i>Check integrity</i>"]
    end
    
    subgraph Decode Stage
        D["Decode<br/><i>C++ service</i><br/><i>Extract frames</i>"]
    end
    
    subgraph Analysis Stage
        A["Analyze<br/><i>Python/ML</i><br/><i>Scene detection</i>"]
    end
    
    subgraph Encode Stage
        E1["Encode 1080p<br/><i>H.264</i>"]
        E2["Encode 4K<br/><i>H.265</i>"]
        E3["Encode 4K<br/><i>AV1</i>"]
    end
    
    subgraph Package Stage
        P["Package<br/><i>HLS/DASH</i>"]
    end
    
    CDN["Upload to CDN"]
    QM["Quality Metrics<br/><i>Parallel pipeline</i>"]
    
    Input --"1. Raw file"--> V
    V --"2. Validated"--> D
    D --"3. Raw frames"--> A
    A --"4. Analysis data"--> E1
    A --"4. Analysis data"--> E2
    A --"4. Analysis data"--> E3
    A -."Tee: Quality analysis".-> QM
    E1 --"5. Encoded stream"--> P
    E2 --"5. Encoded stream"--> P
    E3 --"5. Encoded stream"--> P
    P --"6. Packaged"--> CDN

Netflix’s production encoding pipeline showing language-specific filters (C++ for decode performance, Python for ML analysis) and parallel encoding to multiple formats. The analyze filter uses a tee pattern to feed both the encoding pipeline and a separate quality metrics pipeline without blocking.

---

Interview Essentials

Mid-Level

Explain the basic pattern: filters transform data, pipes connect them. Describe a simple example like log processing (parse → filter → aggregate). Discuss how you’d implement a filter interface in your preferred language. Explain backpressure at a conceptual level—what happens when a slow filter can’t keep up. Describe error handling: retry, skip, or fail the pipeline. Show you understand the trade-off between modularity and performance overhead.

Senior

Design a complete pipeline for a real system (video transcoding, ETL, image processing). Justify filter boundaries—why split here but not there? Discuss scaling strategies: which filters need more instances and why. Explain how you’d implement backpressure using queues, rate limiting, or dynamic scaling. Describe monitoring: what metrics matter (throughput per filter, queue depth, error rates). Compare in-process pipes (channels, queues) vs distributed pipes (Kafka, RabbitMQ). Discuss idempotency: how do you handle retries without duplicate processing? Explain how you’d version filters when the data format changes.

Staff+

Architect a multi-tenant pipeline where different customers can register custom filters. Discuss isolation, security, and resource quotas. Design for evolution: how do you migrate from v1 to v2 filters without downtime? Explain the trade-offs between push (filters actively send data) vs pull (filters request data) models. Discuss exactly-once semantics: how do you guarantee no data loss or duplication across filter failures? Compare pipes and filters to alternatives (monolithic processing, serverless functions, stream processing frameworks). When would you choose each? Explain how you’d implement dynamic pipeline reconfiguration—changing filter order or adding filters at runtime without stopping the system. Discuss cost optimization: how do you balance filter granularity against infrastructure overhead?

Common Interview Questions

How do you handle a filter that needs to look ahead or look back at multiple records?

What happens when a filter crashes mid-processing? How do you recover?

How do you implement transactions across multiple filters?

When would you choose pipes and filters over a monolithic function?

How do you debug a pipeline when a record produces wrong output?

How do you handle schema evolution when filter inputs/outputs change?

Red Flags to Avoid

Can’t explain the difference between a filter and a pipe

Suggests sharing state between filters through global variables

Ignores backpressure until prompted

Can’t articulate when NOT to use this pattern

Designs filters that are too granular or too coarse without justification

Doesn’t consider error handling or monitoring

---

Key Takeaways

Pipes and Filters decomposes complex processing into independent, reusable filters connected by pipes. Each filter does one transformation well, enabling parallel execution and independent scaling.

The pattern trades 20-50% performance overhead (from serialization) for massive gains in flexibility, maintainability, and scalability. Use it when processing logic changes frequently or different stages have different resource needs.

Backpressure handling is critical: slow filters must not crash the pipeline. Use bounded queues, rate limiting, or dynamic scaling to handle mismatched throughput between stages.

Error handling requires strategy: retry transient failures, route permanent failures to dead-letter queues, and ensure the pipeline continues processing other records. Never let one bad record block the entire pipeline.

Real-world implementations (Netflix encoding, Facebook image processing) show the pattern shines for multi-stage transformations where different teams own different stages and independent scaling is essential.

---