Skip to main content

Step 1: Prepare your data

Preparing your dataset before anonymization is a crucial step to ensure high-quality results. This step has the most significant impact on the quality of the generated dataset. Since the anonymization process itself is relatively short, we recommend dedicating most of the project time to data preparation.

This guide outlines the necessary and recommended steps to structure your data for optimal anonymization. Some preprocessing steps can be performed directly within the tool.

Required Preprocessing Steps​

To ensure compatibility with the Avatar anonymization process, your dataset must adhere to the following requirements:

  • Handle Direct Identifiers: Names, Social Security Numbers, email addresses and other direct identifiers must not be anonymized as regular variables. Configure pseudonymization for these columns: they are set aside before anonymization and replaced with substitutes in the output. Keep primary keys and foreign keys and declare them as identifiers in multi-table setups: the platform manages them and preserves the links between tables. Remove only the identifier columns you do not need in the output.
  • Supported File Formats: Provide data in Parquet, CSV or Pandas DataFrame format. If using CSV, ensure it is comma-separated.
  • Standardize Date Format: Convert all date values to YYYY-MM-DD format.
  • Standardize Missing Format: Convert all missing values to the same format.
  • Structured Data Only: The method supports structured, tabular data. Free text, audio, or video content should be removed or transformed beforehand.

While not mandatory, the following best practices enhance anonymization quality:

  • Remove Computed Variables and recompute afterwards: Avoid derived variables that can introduce redundancy and impact anonymization quality. (e.g., BMI calculated from weight & height)
  • Remove redundant information: If columns contain duplicated or derivable information, keep only one of them. For example, if you have both "city" and "region" columns, remove the region since it can be inferred from the city name.
  • Express Relationships as Differences: Where relevant, express relationships between variables as differences rather than absolute values. For example, an end date should be expressed as the number of days since a start date to ensure the chronological consistency.
  • Transform Low-Cardinality Numeric Columns: Convert numeric columns with fewer than 20 unique values into categorical variables.
  • Regroup unique modalities: Convert all unique modalities of a categorical variable into a single to avoid projection distortion and prevent from membership inference attack weakness.
  • Ensure you don't have mixed datatypes: When dealing with a mixed data types variable (e.g., "< 0", 1, 2, 3), the method will automatically consider it as a categorical variable. Please convert to the correct data type beforehand.
  • Split high-dimensional datasets: When working with datasets containing many continuous variables (more than 200), consider splitting the anonymization into multiple smaller batches of correlated variables to improve quality.
  • Split datasets with heterogeneous populations: If your dataset combines two or more distinct groups of individuals, consider splitting it into separate batches (each with a sufficient number of records) to improve modeling quality during anonymization.

The Avatar solution includes built-in tools, called Processors, to assist with these preprocessing steps. You can configure the following processors:

Configurable Processors (optional, for specific use cases):

Automatic Processors (applied transparently without configuration):

  • Automatic Processing — Learn about automatic data handling: missing data, datetime conversion, type preservation, decimal precision, skewed variable transformation, and more

Multi-Table Recommendations:

  • Merge one-to-one relationships: When two tables have a one-to-one relationship, consider joining them into a single table to simplify the schema.
  • Split large heterogeneous tables: If a table combines multiple sources of information, split it into separate tables with clear purposes. For example, divide a comprehensive "visit" table into separate "patient" and "visit" tables.
  • Choose linkage parameters carefully: Select linkage parameters based on the cross-table relationships you need to preserve in the anonymized data.

Specific Preprocessing for Relational Databases and Time Series​

For use cases involving relational databases or time series (i.e., when multiple records exist per individual), tables must adhere to the following structural constraints:

Each table must contain at least one identifier variable, known as a primary key, which holds a unique value for each row.

Relationships between tables are established through foreign keys, which reference the primary key of another table. Unlike primary keys, foreign keys do not need to be unique within their table, allowing multiple rows to link to the same referenced entity.

The more horizontally structured a dataset is (i.e., the higher the ratio of the number of variables to the number of records), the more likely it is that statistical properties will be affected at an equal privacy level. This is particularly relevant for datasets with few individuals but many variables. To mitigate this, we recommend including only the variables necessary for the identified use case rather than attempting to anonymize a dataset intended for multiple use cases simultaneously.

Next Steps: Run the Anonymization​

Once your data is prepared, proceed to the Run section to execute the anonymization process. Keep in mind that optimizing an anonymization process is iterative. After evaluating the anonymized results, you may refine the preparation step to further improve the data quality and privacy balance.