Skip to main content

Security

We consider security as a critical requirement for all our features. We follow security-by-design practices.

You can find a security-oriented architecture diagram below. For more details, see Architecture

Dashed: the optional database connector, only needed for database runs. It is off by default.

Confidentiality-wise:

  • All traffic transits over an encrypted transport (HTTPS). We use Nginx to provide best-in-class security (configurable TLS certificate management, memory safety, hardened TLS stack).
    • As of writing, Nginx terminates HTTPS, which means that traffic between Nginx and the API server uses HTTP. This can be switched to HTTPS if necessary.
    • For On-premise deployment, you can provide your own TLS certificates.

In order to allow handling of very large datasets, data uploads and downloads are done through dedicated S3 endpoints. A dedicated authorization request has to be done for both reading and writing data on that endpoint.

  • All files (original and avatar datasets) are kept for the configured retention period (TTL) and automatically deleted thereafter.

User authentication and authorization​

All user actions are authenticated, stored and can be audited.

  • Authentication requires a valid username and password. Password are stored using a secure mechanism (secure hash algorithm with a unique salt for each password + a secret pepper for the whole codebase).
  • We use JWT for the credentials mechanism. They are set to expire after a configurable amount of time, by default 1 hour. For more information about JWT, checkout jwt.io.
  • We support two roles: Admin and User.

Typical usage data flows​

Here's a typical data flow:

  1. The user logs into the API (using a straight HTTPS client or via one of our client libraries) and receives a temporary JWT for subsequent requests.
  2. The user uploads the datasets (CSV, parquet) containing pseudonymized data to the S3 Storage endpoints, usually via one of our client libraries (web, python). This transfer is encrypted in transit through HTTPS.
  3. The file is stored in storage. The file will be deleted after the configured retention period (TTL).
  4. The user can run multiple anonymization jobs on a given dataset. They can download the avatar dataset (again, through HTTPS) on their machine to conduct further analysis.
    • The avatar dataset will be deleted after a configurable amount of time (see above).

A database run reads its data from a database instead of a file:

  1. The user picks a source and a target among the connections stored by Avatar. The job carries their names, never a password.
  2. The worker hands a short-lived token to the database connector, which reads the source tables and writes them straight to the object storage of the job. The rows do not go through the API.
  3. The anonymized tables are written into the target database, through the connector. Nothing is left to download, and the files of the job follow the usual retention period.

Database connections​

Database runs need credentials for your databases. They are handled so that no job, no worker and no user can read them back:

  • Write-only credentials. A connection's password is sent once, then never returned: not by the API, not by the web client, not in a job. A listing shows the host and the database name only.
  • Encrypted at rest. The API stores each connection encrypted, with a key kept outside its database, so a database dump alone opens nothing. The key is either a Fernet key from the deployment secrets, or a key in a KMS-compatible key manager (AWS KMS, Scaleway Key Manager) that the API never holds. With KMS, each connection is bound to its organization and its name: a value copied into another row is refused.
  • Scoped. A connection belongs to one user, to one organization, or to the deployment. See Connection scopes.
  • Workers never hold a password. A job carries the name of a connection and a token signed by the API, valid for that connection, that user, and a limited time (6 hours by default).
  • One component holds the drivers. Only the database connector opens your databases. It asks the API for the connection string on an internal port that the public port does not serve, that no Ingress refers to, and that a network policy limits to the connector. It keeps the string in memory for 30 seconds and writes nothing to disk.
  • No query endpoint. The connector lists a schema, reads tables, writes into existing tables of a target, and deletes the rows of the tables that a run names. It never drops a table, and runs no query that a user writes. Its error messages give a category, never a host or a credential.
  • Contained network. The connector is never exposed outside the cluster. Its network policy lets it reach DNS, the API's internal port, the object storage, and the public internet, but not the cluster's private networks, the platform's database or the cloud metadata service. A database on a private network must be allowed explicitly, network by network.

On your side, give the source connection a database role that can only read, give the target a role that can only write and delete rows, and never point a target at a production database.

Network flows​

Users reach the platform on a single inbound port, 443 (HTTPS), for the web client, the API, single sign-on and dataset transfers. The outbound flows are listed in Requirements. When the database connector is enabled, it also opens connections to the databases that your users connect to. The optional ones, and exactly what they send, are described in Data sent to Octopize. All of them can be turned off for an air-gapped deployment.

Security limits​

The following global limits can be configured to protect the API:

  • Max request body size: 1 GB (for dataset upload)
  • Global rate limit: 1 request per 100 ms per client per endpoint

What is installed on the user endpoints?​

The end-user will only install a client library for their programming language (e.g. Python) to streamline Avatar API usage.

Those libraries are open source and are audited using static analysis:

What to backup?​

Back up the PostgreSQL databases (Avatar's and Authentik's: users and metadata) and the deployment secrets, including the storage encryption key and, if you use database runs, the key that encrypts the stored connections (databaseConnections.encryptionKeys). Without that key, every stored connection is unreadable and must be entered again. See Backup.

The file storage is temporary and not meant to keep data. Once anonymized, the avatar dataset needs to be downloaded and stored somewhere else.

Development practices​

We follow best practices when it comes to building secure-by-design software:

  • Code reviews
  • Static code analysis (using bandit)
  • Automated dependency analysis (using dependabot and quay.io)
  • Continuous integration
  • Automated and manual testing (we monitor code coverage)
  • Continuous developer training.
  • Threat monitoring
  • Our Github organization requires 2FA and commit signing.