Modern CG pipelines are distributed: artists work across different machines, render farms process thousands of frames in parallel, and pipeline tools need to coordinate dozens of moving parts at once.
A compositor needs to know when a shot finishes rendering. Every downstream process should react immediately when an asset gets approved. The supervisor should not have to find out three hours later when a render fails at frame 847.
Yet many studios still rely on systems that check for updates on a timer, refresh dashboards manually, or depend on an artist to ping a colleague when a task is done, introducing human error into places where automation should be doing the work.
This article covers five real-time communication patterns: polling, webhooks, WebSockets, Server-Sent Events (SSE), and WebRTC. For each one, you will learn what it is, how it applies to a CG pipeline context, and where it helps or hurts. By the end, you will have a clear decision matrix for choosing the right technology for the right problem.
Why Real-Time Matters for CG Pipelines
Real-time communication is not a luxury in production pipelines.
Hardware utilization depends on fast feedback loops. A render farm sitting idle because downstream jobs are waiting on a status update that has not arrived yet is a direct cost. If a compositing job cannot start until it knows that all render layers are complete, and that notification takes five minutes to arrive due to polling lag, you lose five minutes of potential throughput per shot. Across hundreds of shots and multiple farms, it can quickly compound into days of wasted capacity.
Artist iteration speed is also constrained by tool responsiveness. When an artist submits a cache job and has to manually refresh a panel to check progress, they lose flow and context. Tools that push updates instantly let artists stay in context, catch failures early, and queue follow-up work without switching mental gears.
Lastly, automated pipelines break down without reliable event propagation. A pipeline where step B should trigger automatically after step A completes is only as reliable as what connects them. If that connection is fragile, slow, or lossy, the automation breaks and humans have to fill the gap. Real-time communication technologies are the connective tissue of automated pipelines, and choosing the wrong one introduces subtle bugs that are expensive to fix. This is why we wrote this article.
1. Polling
Polling is the simplest way to implement a real-time system: a client repeatedly asks a server "do you have anything new for me?" on a fixed interval.
The client sends a request, the server responds with the current state or any updates, and the client waits until the next interval to ask again.
In a CG pipeline, a typical example is a render watcher script that queries a render management API every 30 seconds to check whether a job has changed status.
The script does not wait for the farm to reach out. It proactively asks, parses the response, and updates a local dashboard or triggers a downstream action.
Pros for CG Pipelines
- Polling is easy to implement. Any tool that can make an HTTP request can poll, which means it works with virtually every pipeline language and framework without special infrastructure. It's a practical choice for small studios or quick internal tools where the cost of standing up event-driven infrastructure is not justified.
- Polling is also stateless on the server side. The server does not need to maintain a persistent connection or keep track of who is listening. For render management APIs that are already built as REST services, polling requires no server changes at all.
Cons for CG Pipelines
- Polling is inefficient. Because the client checks on a schedule regardless of whether anything has changed, the majority of requests return no new information. On a quiet night where no jobs finish for two hours, a watcher script makes hundreds of unnecessary requests. At scale, this adds unnecessary load to APIs that are already under pressure.
- Polling also introduces latency proportional to the interval. A 30-second polling interval means a job completion can take up to 30 seconds to be acted upon. For pipelines where downstream jobs are queued immediately after upstream completion, this delay is baked into every hand-off. Shorter intervals reduce this lag but increase request volume, making the tradeoff hard to tune at scale.
2. Webhooks
A webhook inverts the polling model for machine-to-machine communication. Instead of a web server asking another server repeatedly, the server calls the client when something happens. The client registers a URL endpoint (webhook endpoint), and the server posts a notification to that URL the moment a relevant event occurs.
In a CG pipeline, a practical example is a publish system that fires a webhook to a Slack bot or a pipeline orchestrator the moment an asset is published to the right status. When a lighter approves a lookdev asset and the asset management system records that approval, it immediately sends an HTTP POST to the orchestrator's webhook endpoint with the asset ID and new status. The orchestrator then triggers the downstream lighting build without any polling in the loop.
Pros for CG Pipelines
- Webhooks are event-driven, meaning there is no wasted traffic. The server only sends data when something actually happens, and the client only does work when it receives a call. For pipelines where events are relatively infrequent but must be acted on quickly, this is a significant efficiency gain over polling.
- Webhooks also decouple services cleanly. The asset management system does not need to know what the orchestrator does with the notification; it just fires and forgets. This makes it easier to add new consumers of pipeline events without modifying the source system, which matters in studios where different tools are owned by different teams.
Cons for CG Pipelines
- Webhooks require the receiving service to be publicly reachable or at least network-accessible from the sender. In studio environments with complex network topologies, VPNs, and firewalls, this is not always straightforward. A render farm running in a private subnet cannot easily fire webhooks to a developer's local machine during testing.
- Webhooks are also fire-and-forget by default. If the receiving endpoint is down when the event fires, the notification is lost unless the sending system implements retry logic with exponential backoff and a dead-letter queue. Building that reliability layer adds complexity. For a pipeline that cannot afford to miss a job completion event, a bare webhook integration is not enough without that infrastructure behind it.
3. WebSockets
WebSockets establish a persistent, bidirectional connection between a client and a server. Unlike a regular HTTP, which closes the connection after each request-response cycle, a WebSocket connection stays open. Either side can send messages at any time without waiting for the other to initiate.
In a CG pipeline, WebSockets are a natural fit for live render monitoring dashboards. When an artist or supervisor opens a shot progress page, the browser establishes a WebSocket connection to the render management backend. As frame completions, errors, and job state changes happen on the farm, the server pushes those updates directly to the open connection. The dashboard updates in real time without the browser ever making a new request.
This is what we use to handle events in Kitsu.
Pros for CG Pipelines
- WebSockets eliminate polling overhead while providing lower latency than webhooks for interactive tools. Because the connection is persistent, there is no TCP handshake cost per update. For dashboards that display dozens of concurrent jobs and need to reflect state changes within milliseconds, WebSockets provide the responsiveness that polling cannot.
- The bidirectional nature of WebSockets also enables interactive pipeline tools. A submission UI where an artist configures a render job, submits it, and immediately sees live progress feedback in the same interface becomes practical with WebSockets. The client can send job parameters and receive a stream of status updates over the same connection without switching protocols.
Cons for CG Pipelines
- WebSockets introduce stateful connections that the server must manage. If a render management server is handling thousands of concurrent WebSocket connections from farm nodes and monitoring dashboards, the memory and file descriptor overhead becomes a real engineering concern. Scaling a WebSocket server horizontally requires a shared message broker like Redis or Kafka to fan out messages across instances, which adds infrastructure complexity.
- WebSockets are also more fragile than HTTP in environments with aggressive proxies or load balancers. Some older studio network infrastructure does not handle WebSocket upgrades correctly, causing connections to drop silently or fail entirely. Reconnection logic on the client side is essential but is often underimplemented in standard libraries.
4. Server-Sent Events (SSE)
Server-Sent Events (SSE) is a simpler alternative to WebSockets for cases where the communication is one-directional: the server pushes updates to the client, but the client does not need to send messages back. SSE uses a standard HTTP connection that the server keeps open, streaming events as they occur in a simple text-based format.
In a CG pipeline, SSE is well suited for log streaming. When an artist kicks off a simulation and wants to watch the solver output in real time, an SSE endpoint on the pipeline server can stream each log line as it is written. The browser opens a single HTTP connection and receives a continuous stream of text without polling, without WebSocket complexity, and without any special client library.
Pros for CG Pipelines
- SSE is significantly simpler to implement than WebSockets for server-to-client use cases. It works over plain HTTP, which means it passes through most proxies and load balancers without configuration changes. For a studio where internal tooling is built on standard web stacks, adding an SSE endpoint to an existing API service is straightforward.
- SSE also handles reconnection automatically. The browser's native EventSource API reconnects if the connection drops and resumes from the last received event ID if the server supports it. For a log streaming tool that needs to survive a momentary network hiccup without losing output, this built-in reliability is valuable without any custom code.
Cons for CG Pipelines
- SSE is unidirectional. If the client needs to send data back to the server during the stream, it has to make a separate HTTP request on a different channel. For simple log viewers this is fine, but for interactive tools SSE can lag behind and a WebSocket connection is more appropriate.
- SSE is also limited by the browser's connection pool in some environments. Older browsers cap the number of concurrent SSE connections to the same origin at six, which can become a problem if a pipeline dashboard opens multiple independent SSE streams simultaneously. In practice this is rarely a blocking issue for internal tools, but it is worth knowing before designing a system that relies on many concurrent streams from a single page.
- SSE also doesn't handle binary messages natively. You can of course encode binary files in text, but the size overhead is significant compared to a Websocket equivalent.
5. WebRTC
WebRTC (Web Real-Time Communication) is a peer-to-peer communication protocol designed for low-latency, high-throughput data transfer directly between clients, bypassing the server for the data channel itself. It is best known for video and audio streaming in browser applications but also supports arbitrary binary data transfer through its data channels.
In a CG pipeline, WebRTC has niche but powerful use cases like live preview streaming, chat, or P2P transfers. When an artist is iterating on shading or lighting in a remote GPU workstation and wants to see the viewport rendered output on a local browser without round-tripping through a server, a WebRTC data channel can stream the pixel buffer directly from the GPU machine to the browser with single-digit millisecond latency. This is what most web conferencing solutions use.
Pros for CG Pipelines
- WebRTC offers the lowest achievable latency for browser-based streaming because the data travels peer-to-peer rather than through an intermediate server. For interactive viewport streaming where the user expects the image to update in response to their mouse movement without perceptible lag, WebRTC is the only browser-native option that meets that bar.
- WebRTC also uses efficient binary encoding and built-in congestion control, which makes it well suited for streaming large image data under variable network conditions. The protocol adapts its bitrate dynamically, which is better than a naive WebSocket implementation that sends full-resolution frames regardless of available bandwidth.
Cons for CG Pipelines
- WebRTC is probably the most complex pattern on this list to implement and operate. Establishing a peer-to-peer connection requires a signaling server to exchange session descriptions and network candidates, and often requires a TURN relay server for environments where direct peer connections are blocked by NAT or firewall rules. Studio networks, with their VPNs, proxies, and segmented subnets, are exactly the kind of environment where direct peer connections frequently fail and TURN servers become mandatory.
- WebRTC is also designed primarily for browser environments, though client libraries are available on server programming languages. If the consumer of the data stream is another server process or a native application rather than a browser, WebRTC adds complexity without delivering its main benefit of peer-to-peer efficiency. For those cases, a simpler binary WebSocket stream is usually a better fit.
Conclusion
The five technologies covered in this article sit at different points on a spectrum:
- Polling is the starting point: easy to implement, universally compatible, but wasteful and slow. It is appropriate for low-frequency checks where real-time precision is not required, like a nightly report generator that checks whether all dailies have been rendered.
- Webhooks are the right tool when a server needs to notify one or more external services about discrete events without maintaining a persistent connection. Asset approval workflows, publish triggers, and cross-system integrations are natural fits.
- WebSockets are useful when you need low-latency, bidirectional communication in an interactive tool, like a live render dashboard that also accepts user commands to pause or reprioritize jobs.
- SSE is a simpler alternative when the server needs to push a continuous stream of data to a client and the client does not need to respond, like a log tail viewer or a progress feed for a long-running pipeline process.
- WebRTC is the solution when you need peer-to-peer data transfer with the lowest possible latency, specifically for interactive viewport or preview streaming to a browser.
None of these patterns is universally superior: the studios that build the most responsive and reliable pipelines are the ones that understand each tool well enough to match it precisely to the problem in front of them.
Start by auditing one place in your current pipeline where artists are waiting for information that a system already has. Pick the pattern that fits, implement it, and measure the difference.
