Skip to main content

​
Overview

The Formal Connector operates by reverse-proxying the S3 API, effectively serving as an intermediary that forwards requests to S3. By reverse-proxying all buckets that the configured IAM role has access to, it enables precise control over bucket access and operations. Users can specify the target bucket when using the AWS S3 CLI/API, ensuring that access is streamlined and secure.

​
Creating an S3 resource

Using the S3 Connector requires a hostname to be configured. See Hostname and TLS Configuration for more details.
In Formal, S3 resources represent top-level S3 endpoints that are capable of handling a ListBuckets API call. For example, for AWS-hosted S3, you should use s3.amazonaws.com as your resource hostname. For Cloudflare R2, you should use <ACCOUNT_ID>.r2.cloudflarestorage.com.

​
Authentication

S3 Resources do not support Native Users. For S3, the Connector automatically uses AWS credentials provided via the Instance Metadata Service (IMDS), environment variables, or Pod Identity. See the AWS S3 documentation for more details on S3 authentication.
The IAM role provided to the connector must allow all S3 operations that a user may target while interfacing with the API.

​
Autodiscovery of S3 buckets

The Formal control plane can autodiscover S3 buckets in your AWS account. Please see Cloud Integrations to enable autodiscovery for a given account.

​
Connecting to S3 via the Connector

​
Using the AWS CLI

To connect to S3 via the Connector, you should use your Formal Credentials as your AWS Credentials and modify the endpoint url to the Connector hostname as follows: 
export AWS_ACCESS_KEY_ID="<formal_username>"
export AWS_SECRET_ACCESS_KEY="<formal_access_token>"

aws s3 --endpoint-url <connector_hostname> ls

​
Using the Formal Desktop App

The Formal desktop app streamlines S3 access by seamlessly routing S3 CLI commands to the connectors that can best access the requested AWS account and bucket. This feature requires S3 bucket autodiscovery to be enabled for all AWS accounts that host buckets that users might access via the desktop app. Examples of commands via the desktop app are below. 
# ListBuckets
formal s3 --account-name dev ls
formal s3api --account-id 123456789012 list-buckets

# ListObjectsV2, GetObject, all other bucket-level operations
formal s3 ls s3://my-bucket
formal s3api list-objects-v2 --bucket-name my-bucket

​
Using the Formal S3 Browser

The Formal S3 Browser is a web interface embedded in the connector that allows you to browse, upload, and download files from your S3 buckets. To use the S3 Browser, you can navigate to https://<connector_hostname>:<port>/formal-s3-browser in your web browser. You will then be redirected to the Formal control plane which will automatically authenticate you to the connector and make the S3 Browser available to you. After logging in, you’ll first be presented with a list of the S3 buckets accessible to your connector. S3 Browser Buckets View Clicking on a bucket will reveal the files and folders within it. You can also browse the contents of a folder or upload new files or folders. S3 Browser Files View S3 API traffic from the S3 Browser is proxied through the connector. This ensures that all accesses are logged to the Formal Control Plane and that all policies that apply to S3 API traffic are enforced. For example, when a masking policy is applied to GetObject requests, the S3 Browser will also mask the contents of the file on download. S3 Browser Masked File Download

​
End-user identity propagation

The S3 connector supports end-user identity propagation for internal applications and BI tools. For S3, the connector relies on the HTTP header X-Formal-End-User-Identity to identify the end user by their email. Examples of configuring this for the AWS SDK for various languages are below.
enduser.go
package main

import (
	"context"
	"fmt"
	"log"
	"net/http"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

type customTransport struct {
	base http.RoundTripper
}

func (t *customTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	req.Header.Set("X-Formal-End-User-Identity", "example@example.com")
	return t.base.RoundTrip(req)
}

func main() {
	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("idp:formal:machine:example", "MACHINE_USER_ACCESS_TOKEN", "")),
		config.WithBaseEndpoint("https://demo-env-starter.maytana.connectors.joinformal.com"),
		config.WithHTTPClient(&http.Client{
			Transport: &customTransport{
				base: http.DefaultTransport,
			},
		}),
	)
	if err != nil {
		log.Fatalf("unable to load SDK config, %v", err)
	}

	client := s3.NewFromConfig(cfg)

	output, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
		Bucket: aws.String("s3-demo-bucket-formal"),
		Key:    aws.String("file.txt"),
	})
	if err != nil {
		log.Fatalf("failed to get object, %v", err)
	}

	fmt.Printf("output: %v", output)
}

​
Policy Evaluation

Formal supports the following policy evaluation stages for S3:
  • Session: Evaluate and enforce policies at connection time
  • Pre-Request: Evaluate and enforce policies before request execution

​
Rate-limiting access to buckets

The Formal policy engine supports enforcing rate limits on accesses to S3 buckets based on access counts for a given user over the last minute or hour. An example of such a policy is below. 
package formal.v2

import future.keywords.if

pre_request := { "action": "block", "type": "block_with_formal_message" } if {
  input.bucket.name == "s3-demo-bucket-formal"
  input.bucket.access_count.last_minute > 3
}
Note that we strongly recommend deploying the connector on Kubernetes if enforcing rate limits in a high-availability (multi-instance) deployment. This is because Kubernetes offers optimal support for creating an etcd cluster across connector instances to ensure that each has an up-to-date count of bucket accesses.