Manage Pipeline Resources on BYOC and Dedicated Clusters

Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance.

Prerequisites

  • A running BYOC (not BYOVPC) or Dedicated cluster.

  • An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the benchmark processor.

Understanding compute units

A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or 100m) and 400 MB (400M) of memory.

For sizing purposes, one compute unit supports an estimated message throughput of 1 MB/s. However, actual performance depends on the complexity of a pipeline, including the components it contains and the processing it does.

You can allocate a maximum of 72 compute units per pipeline. You can add compute units in increments of one up to 15 compute units. Beyond this, scaling options increase to 33 and then to 72 compute units. This scaling strategy is based on the number of machine cores required to provision resources, which scale from two to four, and then to eight cores.

Server resources are charged at an hourly rate in compute unit hours (compute/hour).

Number of compute units CPU Memory

1

0.1 CPU (100m)

400 MB (400M)

2

0.2 CPU (200m)

800 MB (800M)

3

0.3 CPU (300m)

1.2 GB (1200M)

4

0.4 CPU (400m)

1.6 GB (1600M)

5

0.5 CPU (500m)

2.0 GB (2000M)

6

0.6 CPU (600m)

2.4 GB (2400M)

7

0.7 CPU (700m)

2.8 GB (2800M)

8

0.8 CPU (800m)

3.2 GB (3200M)

9

0.9 CPU (900m)

3.6 GB (3600M)

10

1.0 CPU (1000m)

4.0 GB (4000M)

11

1.1 CPU (1100m)

4.4 GB (4400M)

12

1.2 CPU (1200m)

4.8 GB (4800M)

13

1.3 CPU (1300m)

5.2 GB (5200M)

14

1.4 CPU (1400m)

5.6 GB (5600M)

15

1.5 CPU (1500m)

6.0 GB (6000M)

33

3.3 CPU (3300m)

13.2 GB (13200M)

72

7.2 CPU (7200m)

28.8 GB (28800M)
For pipelines with embedded Ollama AI components, one GPU is automatically allocated to the pipeline, which is equivalent to 30 compute units, or 3.0 CPU (3000m) and 12 GB of memory (12000M).

Set an initial resource limit

When you create a data pipeline, you can allocate a fixed amount of server resources to it using compute units.

If your pipeline reaches the CPU limit, it becomes throttled, which reduces the data processing rate. If it reaches the memory limit, the pipeline restarts.

To set an initial resource limit:

  1. Log in to Redpanda Cloud.

  2. On the Clusters page, select the cluster where you want to add a pipeline.

  3. Go to the Connect page.

  4. Select the Redpanda Connect tab.

  5. Click Create pipeline.

  6. Enter details for your pipeline, including a short name and description.

  7. For Compute units, leave the default 1 compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate a maximum of 72 compute units.

  8. For Configuration, paste your pipeline configuration and click Create to run it.

Scale resources

View the server resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption.

To view resources already allocated to a data pipeline:

  1. Log in to Redpanda Cloud.

  2. Go to the cluster where the pipeline is set up.

  3. On the Connect page, select your pipeline and look at the value for Resources.

    • CPU resources are displayed first, in milliCPU. For example, 1 compute unit is 100m or 0.1 CPU.

    • Memory is displayed next in megabytes. For example, 1 compute unit is 400M or 400 MB.

To scale the resources for a pipeline:

  1. Log in to Redpanda Cloud.

  2. Go to the cluster where the pipeline is set up.

  3. On the Connect page, select your pipeline and click Edit.

  4. For Compute units, update the number of compute units. You can allocate a maximum of 72 compute units per pipeline.

  5. Click Update to apply your changes. The specified resources are available immediately.