Currently, member companies are testing the proposed data provenance standards in various use cases, encompassing areas such as regulatory compliance, supply chain management, procurement processes, and virtual patient healthcare systems. These tests are not only focused on assessing business value and understanding the practical implementation requirements, but also include a crucial component of governance testing. This governance testing is designed to ensure that the standards adhere to relevant policies and legal frameworks, thereby making data management practices transparent, accountable, and aligned with both organizational and regulatory expectations.

We are actively calling for input. (1) You can learn more by watching the video above and/or downloading our info pack.(2) You can take our survey. The survey will be open until January 20, 2024. (3) You can join our open Community of Practice to offer new use cases, share learnings, and refine the work. Our goal is to create a baseline set of data provenance standards that is both valuable and implementable for a wide range of cross-industry traditional data and AI use cases.

No. The data provenance standards are designed for both traditional data and AI use cases. They are designed to bring transparency to the origin of datasets, which are helpful for use cases across the enterprise. Note that the standards currently only apply at the dataset level, not the element, table, or row level.

The data provenance standards address most generative AI use cases, but may not be applicable to all use cases. For example, data provenance standards can be challenging to apply effectively to large language models (LLMs) that are trained on vast amounts of public data sourced from diverse locations on the Internet.

Our goal is to refine the standards—based on both testing and practitioner input—and release a version 1 in early Q2 2024. Timing is subject to change. The standards will be available in a variety of formats, ranging from a PDF checklist to JSON and XML, which allows for automation. The standards and supporting tools will be available without charge.

While we can provide guidance on how enterprises and data providers can effectively implement the standards to ensure consistency, interoperability, and data discoverability, the standards and metadata have been designed with ample flexibility to address diverse needs. A values library will also be available for organizations to use and reference, further enhancing consistency across industries. The library will be flexible to accommodate unique values not previously identified through existing use cases.

We have explored and are considering recommendations around immutable records, such as blockchain and data watermarking. Such mechanisms make it nearly impossible to delete or alter data without the consensus of network participants. This immutability can help protect the integrity of metadata associated with the dataset. However, because blockchain, watermarking, and similar solutions can be technically challenging to adopt and cost-prohibitive, the D&TA is not prescribing a method of immutability at this time.

Yes, the data provenance standards can be integrated with systems designed to support data governance, including Collibra.

The data provenance standards are designed to be cross-industry. They were created by practitioners from healthcare, automotive, IT, media, banking and finance, retail, education and other industries. We encourage practitioners to share new use cases and offer new metadata fields for consideration, and we would welcome any such practitioners to join our Community of Practice.

Yes. Organizations must navigate a complex landscape of laws and regulations to ensure their data practices are compliant. The Legal Rights standard identifies the legal or regulatory framework applicable to the current dataset, along with the required data attributions, associated copyright or trademark, and localization and processing requirements. The Privacy and Protection Standard further supports compliance by identifying if sensitive data is present and if any privacy-enhancing techniques have been applied, thus identifying where further steps must be taken to ensure data security and confidentiality.

We expect the standards to evolve over time, based on the experience of those implementing them. Currently, they are in a testing phase. We will refine the standards—based on both testing and community input —and release a fully accessible version 1 in early Q2 2024. Timing is subject to change.
The standards currently apply at the dataset level. After use of version 1, we imagine the standards will be refined over time and become more applicable at the data cell or row level.

After the testing phase, we plan to share examples or case studies of our learnings and plans for implementation.