Skip to main content

​
Overview

The Formal Connector enables secure access to BigQuery. When domain-wide delegation is enabled, it can impersonate users in your organization, ensuring proper access control and audit logging of all BigQuery operations.

​
Differences with other technologies

The connection to BigQuery differs from other technologies in two important ways:
  • No Native Users: Unlike Postgres or HTTP, Formal native users are not supported for BigQuery currently. Instead, the connector uses Google ADC to connect to BigQuery.
  • Automatic Impersonation: When domain-wide delegation is configured, the Connector will automatically impersonate detected end users for BigQuery requests.

​
Setup

​
1. Configure Domain-wide Delegation

Before creating a BigQuery resource in Formal, you’ll need a service account with domain-wide delegation enabled in your Google Workspace. This step must be done by an admin user of your Google organization. Start by creating a service account in your Google Cloud Console if you don’t have one already, then enable domain-wide delegation:
  • Go to your Google Workspace Admin Console
  • Navigate to Security > Access and data control > API Controls > Manage domain-wide delegation.
  • Click Add New:
    • Client ID: Your service account’s client ID
    • OAuth Scopes:
      • https://www.googleapis.com/auth/bigquery
      • https://www.googleapis.com/auth/cloud-platform
  • Click Authorize

​
2. Create a BigQuery Resource

To create a BigQuery Resource in Formal, follow these steps:
  1. Navigate to the Resources page.
  2. Click on the “Create Resource” button.
  3. Fill in the required fields:
    • Technology: Select BigQuery from the dropdown
    • Resource Friendly Name: Enter a name of your choice to identify this resource
    • Hostname: Enter bigquery.googleapis.com
    • Port: Enter 443 if your Connector has TLS certificates enabled, anything else otherwise (e.g. 7777)
  4. Click on the Create button to complete the resource creation.

​
3. Connect to BigQuery

The last step before querying BigQuery is to deploy your Formal Connector using your service account credentials. The connector uses Google ADC to detect the service account credentials and connect to BigQuery. Once your Formal Connector is set up, you can connect to BigQuery through the Formal Connector using the bq command line tool: 
bq --api ${PROTOCOL}://${HOSTNAME}:${PORT} query "SELECT 1"
Here’s another example of how to connect to BigQuery through the Formal Connector using Python: 
from google.cloud import bigquery  # requires google-cloud-bigquery

PROTOCOL = "..."  # "http" or "https", depending on your connector setup
HOSTNAME = "..."  # Your connector hostname
PORT = "..."  # Your connector listener port

client = bigquery.Client(
    client_options={"api_endpoint": f"{PROTOCOL}://{HOSTNAME}:{PORT}"}
)

query = "SELECT word FROM `bigquery-public-data.samples.shakespeare` LIMIT 10"
for row in client.query(query):
    print(row.word)

​
Policy Evaluation

Formal supports the following policy evaluation stages for BigQuery:
  • Session: Evaluate and enforce policies at connection time
  • Pre-Request: Evaluate and enforce policies before query execution
  • Post-Request: Evaluate and enforce policies after data retrieval

​
Monitoring and Logging

The Connector provides detailed logs of all BigQuery operations, including:
  • Query execution
  • Data access patterns
  • User identity
  • Evaluated / triggered policies
These logs can be accessed through the Formal Console for audit and compliance purposes.

​
Troubleshooting

Common issues include:
  • TLS errors: Google SDK is strict when using a TLS certificate, and will fail if it isn’t signed by a trusted certificate authority.
  • Service account credentials: The Formal Connector doesn’t use the correct service account credentials
  • Permissions: The service account does not have the necessary permissions, especially domain-wide delegation
If you feel stuck, please reach out to us on Slack!