Skip to main content
For load balancing and redundancy, we recommend deploying multiple Connector instances to represent the same logical Connector. The Connector instances can operate entirely independently, but you may want to share their state across all instances to enable advanced features such as distributed rate limiting or BigQuery job ID caching. To do this, you will need to enable these instances to form a cluster.

​
Configuration Requirements

​
Discovery

To enable clustering, the Connectors need to be able to discover each other. Clustering is supported on AWS ECS and Kubernetes.
For AWS ECS deployments, configure IAM permissions to allow Connector instances to list and describe ECS tasks. This enables them to discover other instances in the same service:
# Allow ECS tasks to discover peers for cluster formation
resource "aws_iam_policy" "ecs_task_permissions" {
  name        = "connector-ecs-task-permissions"
  description = "Allow ECS tasks to discover peers for cluster formation"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action   = ["ecs:DescribeTasks", "ecs:ListTasks"],
        Effect   = "Allow",
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_task_permissions" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = aws_iam_policy.ecs_task_permissions.arn
}
See our AWS Terraform example for a complete configuration.

​
Network

Connectors communicate with each other over the network on ports 7946.
Configure the security group to allow inter-instance communication on the cluster ports:
# Allow internal cluster communication on port 7946
resource "aws_security_group" "connector" {
  ingress {
    description = "Cluster communication port TCP"
    from_port   = 7946
    to_port     = 7946
    protocol    = "tcp"
    self        = true  # Allow from same security group
  }
  ingress {
    description = "Cluster communication port UDP"
    from_port   = 7946
    to_port     = 7946
    protocol    = "udp"
    self        = true  # Allow from same security group
  }
  # ...
}
See our AWS Terraform example for a complete configuration.

​
Example

Here’s an example policy that uses the cluster’s shared state to enforce a rate limit on S3 bucket access: 
package formal.v2

import future.keywords.if

pre_request := { "action": "block", "type": "block_with_formal_message" } if {
  input.bucket.name == "sensitive-bucket"
  input.bucket.access_count.last_minute > 5
}
This policy blocks access to sensitive-bucket if it has been accessed more than five times in the last minute by the user, across all Connector instances.
If a Connector instance fails, the cluster automatically rebalances and continues operating. Features that leverage the shared state is preserved across the remaining healthy instances.