Utility
The evaluation of synthetic data quality hinges on a delicate balance between privacy and utility. While privacy ensures that individuals cannot be identified or linked, utility focuses on the preservation of the original dataset's statistical properties, enabling synthetic data to be a reliable substitute for real data.
Evaluation of Utility
To assess the utility of synthetic data, we adopt a framework based on three generic levels of metrics. These levels offer a comprehensive evaluation of how well the synthetic data retains the essential characteristics of the original dataset:
- Univariate Utility: Measures the similarity of individual variable distributions between the original and synthetic datasets.
- Bivariate Utility: Evaluates the preservation of pairwise correlations between variables.
- Multivariate Utility: Assesses the retention of the dataset's global structure, including clustering patterns and overall data relationships.
More details about our privacy metrics here:
While these three levels provide a generic foundation for utility evaluation, we strongly encourage the incorporation of use-case-specific metrics tailored to the needs of the dataset and its intended applications. Custom metrics allow for a more targeted assessment of the synthetic data’s utility by focusing on specific aspects or features crucial to the use case.
The metrics are designed with a minimization objective: lower values indicate better utility, meaning that the synthetic data closely mirrors the statistical properties of the original dataset.
The Importance of Context
Preserving utility is most effective when the desired use of the synthetic data is clearly defined. Adding custom metrics specific to the dataset or its application enhances the evaluation process. For example:
A healthcare dataset may prioritize metrics that assess the retention of diagnostic trends. A financial dataset may focus on the preservation of volatility or correlations between financial indicators.
Utility vs. Privacy: The Balance
Utility must always be evaluated in tandem with privacy to ensure that the synthetic data fulfills its purpose without compromising anonymity. As with privacy, the context of use is critical in determining the adequacy of utility:
For exploratory data analysis, retaining global trends and distributions may suffice. For training machine learning models, the preservation of multivariate structures and complex interactions is often necessary.
By systematically evaluating utility across these three levels and incorporating use-case-specific metrics, our tools empower users to ensure that synthetic data is both privacy-compliant and fit for its intended purpose. This dual focus ensures the creation of synthetic datasets that are secure, reliable, and practical for a wide range of applications.