Skip to main content

Batch Processing

Avatar processes large datasets in batches. A dataset of any size can be anonymized, and the memory a job needs depends on the batch size, not on the size of the dataset.

A dataset smaller than one batch is processed in a single batch. In that case, nothing on this page changes your results.

How it works​

  1. Cut. The dataset is divided into batches of a fixed number of records. The batch size is a setting of the deployment, not a job parameter.
  2. Random assignment. Records are assigned to batches at random. Each batch is a representative sample of the whole dataset, even when the file is sorted by date, identifier or class.
  3. Anonymization. Each batch is anonymized independently. The neighbours used to generate an avatar come from the same batch as the original record.
  4. Assembly. The batches are put back together. The avatars keep the order of the original records, so the 1-for-1 mapping is preserved. The shuffled output is shuffled across the whole dataset, not within each batch.

Multi-table and time series data​

A batch never separates records that must be modeled together:

  • In a multi-table dataset, a parent record and its child records are always in the same batch. This depends on valid primary and foreign keys.
  • In a time series dataset, an individual and all their time series are always in the same batch.

An individual with a very large number of linked records is kept whole. Its batch can then be larger than the configured size.

Metrics​

The dataset checks run on every record. The privacy and utility metrics are computed on the first batch, which is a representative sample. Each avatar is compared with the original records of the batch it was generated from. See the technical assessment report.

What this means for you​

For users:

  • There is no parameter to set. Batch processing applies automatically.
  • The same dataset with the same seed gives the same batches and the same results.
  • With data augmentation, the additional records of a batch are generated from the original records of that batch.

For administrators of a self-hosted deployment:

  • The memory that a job uses depends mainly on the batch size. A larger dataset takes more time, but does not need proportionally more memory.
  • Contact Octopize before you change the batch size: it changes the memory needs and the number of neighbours available to each record.