Skip to main content
For load balancing and redundancy, we recommend deploying multiple Connector instances to represent the same logical Connector. The Connector instances can operate entirely independently, but you may want to share their state across all instances to enable advanced features such as distributed rate limiting or BigQuery job ID caching. To do this, you will need to enable these instances to form a cluster.

​
Configuration Requirements

​
Discovery

To enable clustering, the Connectors need to be able to discover each other. Clustering is supported on AWS ECS and Kubernetes.
  • AWS ECS Fargate (Terraform)
  • Kubernetes (Helm)
For AWS ECS deployments, configure IAM permissions to allow Connector instances to list and describe ECS tasks. This enables them to discover other instances in the same service:
# Allow ECS tasks to discover peers for cluster formation
resource "aws_iam_policy" "ecs_task_permissions" {
  name        = "connector-ecs-task-permissions"
  description = "Allow ECS tasks to discover peers for cluster formation"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action   = ["ecs:DescribeTasks", "ecs:ListTasks"],
        Effect   = "Allow",
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_task_permissions" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = aws_iam_policy.ecs_task_permissions.arn
}
See our AWS Terraform example for a complete configuration.

​
Network

Connectors communicate with each other over the network on ports 2379 and 2380.
  • AWS ECS Fargate (Terraform)
  • Kubernetes (Helm)
Configure the security group to allow inter-instance communication on the cluster ports:
# Allow internal cluster communication on ports 2379 and 2380
resource "aws_security_group" "connector" {
  ingress {
    description = "Cluster client port"
    from_port   = 2379
    to_port     = 2379
    protocol    = "tcp"
    self        = true  # Allow from same security group
  }
  ingress {
    description = "Cluster peer port"
    from_port   = 2380
    to_port     = 2380
    protocol    = "tcp"
    self        = true  # Allow from same security group
  }
  # ...
}
See our AWS Terraform example for a complete configuration.

​
Filesystem

The Connector process requires read and write permissions to the /default.etcd directory. This directory is used by the embedded state server (etcd) to persist cluster state, including member information, configuration, and the distributed key-value store used for features like rate limiting and BigQuery job ID caching.
If the directory is not writable or cannot be created with read/write permissions, the state server will fail to start and features that require clustering will not work.

​
Example

Here’s an example policy that uses the cluster’s shared state to enforce a rate limit on S3 bucket access: 
package formal.v2

import future.keywords.if

pre_request := { "action": "block", "type": "block_with_formal_message" } if {
  input.bucket.name == "sensitive-bucket"
  input.bucket.access_count.last_minute > 5
}
This policy blocks access to sensitive-bucket if it has been accessed more than five times in the last minute by the user, across all Connector instances.
If a Connector instance fails, the cluster automatically rebalances and continues operating. Features that leverage the shared state is preserved across the remaining healthy instances.