Skip to main content
Your Formal Connector deployment will require different resource requirements depending on the specifics of your setup. Factors like the number of active connections you expect to pass through the Connector at once, the policies you configure, the payload sizes of requests and responses, and more can affect the health of individual Connector instances. There are two ways we help our customers get observability into their Connectors: Logs and Metrics.

Logs

The Connector emits logs to stdout/stderr that can be captured in your log collector of choice. For example, if deploying your Connector on ECS Fargate using Terraform, use logConfiguration to route these logs to be sent to your observability tool of choice:
resource "aws_ecs_task_definition" "ecs_task" {
  # ignoring other field irrelevant to logs in this example
  container_definitions = jsonencode([
    {
      name      = var.name
      image     = var.container_image
      essential = true
      environment = [
        {
          name  = "DD_VERSION"
          value = "1.0.0"
        },
        {
          name  = "DD_ENV",
          value = "prod"
        },
        {
          name  = "DD_SERVICE"
          value = var.name
        },
        # specify other env vars here
      ],
      logConfiguration = {
        logDriver = "awsfirelens"
        options = {
          "Name"       = "datadog",
          "Host"       = "http-intake.logs.datadoghq.eu",
          "TLS"        = "on",
          "dd_source"  = var.name,
          "provider"   = "ecs",
          "dd_service" = var.name,
          "apikey"     = var.datadog_api_key
        }
      }
      dependsOn = [
        { "containerName" : "log_router", "condition" : "START" },
        { "condition" = "HEALTHY", "containerName" = "datadog-agent" }
      ]
    },
    {
      name              = "log_router"
      image             = "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable"
      memoryReservation = 50,
      firelensConfiguration = {
        "type" = "fluentbit",
        "options" = {
          "enable-ecs-log-metadata" = "true"
        }
      },
    },
    {
      name  = "datadog-agent",
      image = "public.ecr.aws/datadog/agent:latest",
      portMappings = [
        {
          "containerPort" = 8126,
          "hostPort"      = 8126,
          "protocol"      = "tcp"
        }
        {
          # We need this for metrics
          "containerPort" = 4317,
          "hostPort"      = 4317,
          "protocol"      = "tcp"
        }
      ],
      environment = [{
        "name"  = "ECS_FARGATE",
        "value" = "true"
        },
        {
          "name"  = "DD_APM_ENABLED",
          "value" = "true"
        },
        {
          "name"  = "DD_LOGS_ENABLED",
          "value" = "true"
        },
        {
          "name"  = "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
          "value" = "true"
        },
        {
          "name"  = "DD_APM_NON_LOCAL_TRAFFIC",
          "value" = "true"
        },
        {
          "name"  = "DD_API_KEY",
          "value" = var.datadog_api_key
        },
        {
          "name"  = "DD_SITE",
          "value" = "datadoghq.eu"
      }]
    }
  ])
}

Metrics

The Connector emits metrics over OTLP/gRPC to an OpenTelemetry Collector such as the Datadog Agent to 127.0.0.1 port 4317. The Connector currently emits the following metrics:
  • formal.connector.instance.heartbeat: A heartbeat gauge metric once every 30 seconds to verify Connector instance uptime.
  • formal.connector.memory.usage_percent: A memory usage percentage gauge metric based on the connector’s runtime memory allotment. Note that these percentages may look different than the host’s memory usage if the memory allotted to the Connector is less than the overall host.
  • formal.connector.connections_opened: A counter that is incremented every time the instance accepts a new TCP connection from a client.
  • formal.connector.connections_closed: A counter that is incremented every time the instance closes a TCP connection from a client.
  • formal.connector.active_connections: A gauge metric of the current number of open TCP client connections that the instance has.
  • formal.control_plane.pings_received: A counter metric that is incremented every time we receive a ping from the Control Plane.
  • formal.connector.resource_health_check: A gauge metric of resource health checks.

Deploying an OpenTelemetry Collector

The OpenTelemetry Collector should be running on the same host and listening to OTLP/gRPC on port 4317. For example, you could deploy the Datadog Agent as a container on that same host (so that it is accessible over 127.0.0.1) and explicitly specify the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317 environment variable. The collector should also work well as a separate process on the same host. It is not supported to have the connector send metrics to another hostname and port. If nothing is listening on that port, the connector will refrain from sending metrics entirely.

Datadog Dashboard

To visualize connector health in a single place, we provide a template dashboard for you to import into your Datadog instance.
  • Enter a dashboard name.
  • Choose the New Dashboard button.
  • In the new dashboard, choose the Configure button.
  • Choose the Import dashboard JSON button.
  • Upload the dashboard JSON file downloaded in the previous steps.