...
There are many assessment tools to do a pre-FAIR assessment of your (meta)data. FAIRassist holds a (manually created) collection with of various different tools. These include manual questionnaires or checklists and , as well as automated tests (often only applicable to datasets that are already public and have a persistent identifier (eg. DOI)). The tools help users understand how to achieve a state of "FAIRness", and how this can be measured and improved. Furthermore, a 2022 publication (FAIR assessment tools: evaluating use and performance) compared a number of tools. Of these and the tools listed on FAIRassist, we suggest that the following can be considered for your pre-FAIR assessment:
[Hannah: Fieke pointed out something important; there are basically two kinds of tools for the FAIR assessment. One group assesses (often in a semi-automated way) the FAIRness of (meta)data which already has a persistent identifier (such as a DOI). The other group assesses FAIRness (often in the form of a survey, questionnaire or checklist) of (meta)data without persistent identifier.]
Online self-assessment surveys
These tools allow you to fill in an online form. The result of the survey can be e.g. a score to indicate the FAIRness of your (meta)data. Some tools additionally provide advice on how to improve FAIRness. Well-known online surveys include:
...
Tool
...
Description
...
Quick user guide
...
...
Provided by Australian Research Data Commons, this 12-question online survey provides a visual indication on the FAIRness level of your (meta)data and provides resources on how to improve it.
Online self-assessment surveys
These tools allow you to fill in an online form. The result of the survey can be e.g. a score to indicate the FAIRness of your (meta)data. Some tools additionally provide advice on how to improve FAIRness. Well-known online surveys include:
Tool | Description | Quick user guide | |
|---|---|---|---|
Provided by Australian Research Data Commons, this 12-question online survey provides a visual indication on the FAIRness level of your (meta)data and provides resources on how to improve it. | Fill in the 12 questions in the survey, potentially with help of a FAIR expert/data steward. | ||
Provided by DANS, this online survey gives a FAIRness score. Furthermore, it provides advice on how to improve the FAIRness of your (meta)data. | Fill in the 12 questions in the survey, potentially with help assistance of a FAIR expert/data steward. | ||
Provided by DANS, this online survey gives a FAIRness score. Furthermore, it provides advice on how to improve the FAIRness of your (meta)data. | Fill in the 12 questions in the survey, potentially with help of a FAIR expert/data steward. | ||
The FAIR Data Maturity Model aims to harmonise outcomes of FAIR assessment tools to make these comparable. Based on the FAIR principles The FAIR Data Maturity Model | The FAIR Data Maturity Model aims to harmonise outcomes of FAIR assessment tools to make these comparable. Based on the FAIR principles and sub-principles, they have created a list of universal 'maturity indicators'. Their work resulted in a checklist (with extensive description of all maturity indicators), which can be used to assess the FAIRness of your (meta)data. Their work resulted in a checklist (with extensive description of all maturity indicators), which can be used to harmonise outcomes as well as directly assess the FAIRness of your (meta)data. (??????????????) | Download the excel file from Zenodo and in the ‘FAIR Indicators_v0.05’ tab, give a score to the 41 different ‘maturity indicators’, by selecting the level from the drop-down menu in the ‘METRIC’- column, that fits the status if your (meta)data best. Potentially perform this with help assistance of a FAIR expert/data steward. View the results in the ‘LEVELS' tab. Detailed definitions and examples for all 'maturity indicators’ can be found in the documentation on zenodo. |
Online (Semi-) automated
...
Tool
...
Description
...
Quick user guide
...
...
The FAIR Evaluator assesses each FAIR metric automatically and returns a quantitative score for the FAIRness of the metadata of a resource. It provides feedback on how further improve FAIRness.
The public version of The FAIR Evaluator has been used to assess >5500 datasets.
...
...
“The FAIRshake toolkit was developed to enable the establishment of community-driven FAIR metrics and rubrics paired with manual and automated FAIR assessments. FAIR assessments are visualized as an insignia that can be embedded within digital-resources-hosting websites. Using FAIRshake, a variety of biomedical digital resources can be manually and automatically evaluated for their level of FAIRness.“ (FAIRshake documentation)
The FAIRshake website currently show the results for 132 projects and offers 65 rubics for reuse .
...
The extensive documentation (including YouTube tutorials) can be found here.
...
...
“FAIR-Checker is a web interface to evaluate FAIR metrics and to provide developers with technical FAIRification hints. It's also a Python framework aimed at easing the implementation of FAIR metrics.” (FAIRassist)
FAIRchecker does over 18000 metrics evaluations per month.
...
In the ‘Check' page, paste an URL or DOI and click on 'Test all metrics’. The assessment will run automatically and return a score for 12 FAIR sub-principles. If a sub-principle does not reach the highest score, you can view recommendations on how to improve.
For even more surveys and (semi-) automated tools, see FAIRassist.
FAIR assessment tools vary greatly in their outcomes. The FAIR Data Maturity Model (created by the Research Data Alliance, or RDA) aims to harmonise outcomes of FAIR assessment tools to make these comparable. Based on the FAIR principles and sub-principles, they have created a list of universal 'maturity indicators'. Their work resulted in a checklist (with extensive description of al maturity indicators), which can be used to assess the FAIRness of your (meta)data.
The FAIRplus dataset maturity indicators were created based on previous work around the FAIR indicators, done by the Research Data Alliance (RDA) and the FAIRsFAIR projects:
...
FAIR Data Maturity Model Working Group. (2020). FAIR Data Maturity Model. Specification and Guidelines (1.0). https://doi.org/10.15497/rda00050
...
The FAIRplus dataset maturity indicators were created based on previous work by the Research Data Alliance (RDA, see FAIR Data Maturity Model above) and the FAIRsFAIR projects. This model evaluates FAIRness of data in three categories (Content related, Representation and format, Hosting environment capabilities) and five levels of maturity per category (ranging from Single Use Data to Managed Data Assets). For each category, indicators have been defined to describe the requirements to reach a certain level of maturity in that category. | The spreadsheet used to assess the maturity of your dataset can be found on GitHub. In the 'FAIR-DSM Assessment Sheet v1.2' tab, a pre- and post-FAIR assessment can be performed, potentially with assistance from a FAIR expert/data steward. [Hannah: unfortunately there is not really a user guide, so one has to guess how to fill this in] |
Online (Semi-) automated
Tool | Description | Quick user guide |
|---|---|---|
The FAIR Evaluator assesses each FAIR metric automatically and returns a quantitative score for the FAIRness of the metadata of a resource. It provides feedback on how further improve FAIRness. The public version of The FAIR Evaluator has been used to assess >5500 datasets. | A guide on how to use the FAIR Evaluator can be found in the FAIR Cookbook. | |
“The FAIRshake toolkit was developed to enable the establishment of community-driven FAIR metrics and rubrics paired with manual and automated FAIR assessments. FAIR assessments are visualized as an insignia that can be embedded within digital-resources-hosting websites. Using FAIRshake, a variety of biomedical digital resources can be manually and automatically evaluated for their level of FAIRness.“ (FAIRshake documentation) The FAIRshake website currently show the results for 132 projects and offers 65 rubics for reuse . | The extensive documentation (including YouTube tutorials) can be found here. See for more information also the FAIR Cookbook. | |
“FAIR-Checker is a web interface to evaluate FAIR metrics and to provide developers with technical FAIRification hints. It's also a Python framework aimed at easing the implementation of FAIR metrics.” (FAIRassist) FAIRchecker does over 18000 metrics evaluations per month. | In the ‘Check' page, paste an URL or DOI and click on 'Test all metrics’. The assessment will run automatically and return a score for 12 FAIR sub-principles. If a sub-principle does not reach the highest score, you can view recommendations on how to improve. |
For even more surveys and (semi-) automated tools, see FAIRassist.
FAIR assessment tools vary greatly in their outcomes. The FAIR Data Maturity Model (created by the Research Data Alliance, or RDA) aims to harmonise outcomes of FAIR assessment tools to make these comparable. Based on the FAIR principles and sub-principles, they have created a list of universal 'maturity indicators'. Their work resulted in a checklist (with extensive description of al maturity indicators), which can be used to assess the FAIRness of your (meta)data.
The FAIRplus dataset maturity indicators were created based on previous work around the FAIR indicators, done by the Research Data Alliance (RDA) and the FAIRsFAIR projects:
FAIR Data Maturity Model Working Group. (2020). FAIR Data Maturity Model. Specification and Guidelines (1.0). https://doi.org/10.528115497/zenodo.4081213
In the definitions of the FAIRplus-DSM indicators, you will find a link to the corresponding RDA or FAIRsFAIR indicator when they are related.
...
Devaraju, Anusuriya, Huber, Robert, Mokrane, Mustapha, Herterich, Patricia, Cepinskas, Linas, de Vries, Jerry, L’Hours, Herve, Davidson, Joy, & Angus White. (2020). FAIRsFAIR Data Object Assessment Metrics (0.4). Zenodo. https://doi.org/10.5281/zenodo.4081213
In the definitions of the FAIRplus-DSM indicators, you will find a link to the corresponding RDA or FAIRsFAIR indicator when they are related.
Hannah mentions the Data Maturity Model. This is also here on FAIRplus. There is also this Github from FAIRplus and the sheet for the actual assessment is here. Could be worrying: last update was last year.
...
‘The FAIR Principle R1.3 states that “(Meta)data meet domain-relevant Community standards”. This is the only explicit reference in the FAIR Principles to the role played by domain-specific communities in FAIR. It is interesting to note that an advanced, online, automated, FAIR maturity evaluation system [22] did not attempt to implement a maturity indicator for FAIR Principle R1.3. It was not obvious during the development of the evaluator system how to test for “domain-relevant Community standards” as there exists, in general, no venue where communities publicly and in machine-readable formats declare data and metadata standards, and other FAIR practices. We propose the existence of a valid, machine-actionable FIP be adopted as a maturity indicator for FAIR Principle R1.3.’
[Hannah] There are also these tools [Mijke: these are the ones Nivel used in a recent project - have them write the community example?]:: This might not be applicable to all FAIRification processes, but I think it might be of added value in some cases to assess the current FAIRness also in relation to community standards, so I think it might be nice to include this as well. However, I find it hard to find any information about community standards..]
FIP Mini Questionnaire from GO-FAIR: https://www.go-fair.org/how-to-go-fair/fair-implementation-profile/fip-mini-questionnaire/Data Maturity Model: https://zenodo.org/records/3909563#.YGRNnq8za70
[Mijke: RDMkit has a page on this → https://rdmkit.elixir-europe.org/compliance_monitoring#how-can-you-measure-and-document-data-management-capabilities ]
[Sander]
FAIRCookbook
Assessment Chapter in the FAIRCookbook. It currently has recipes for two tools (no idea how they work yet):
More Checklists and tools:
...
More Checklists and tools:
A Checklist produced for use at the EUDAT summer school to discuss how FAIR the participant's research data were and what measures could be taken to improve FAIRness:
https://zenodo.org/records/1065991#.Xs_XpC2cbOQ%C2%A0 [Hannah; this is also an offline checklist; not sure if we should recommend to consider. I also think it is rather limited compared to the rest of the tools/checklists]
Furthermore: FAIR Evaluator (FAIRopoly and FAIR Guidance) – text copied below.
FAIRopoly
As a task under the objectives of the EJP RD, we created a set of software packages – The FAIR Evaluator – that coded each Metric into an automatable software-based test, and created an engine that could automatically apply these tests to the metadata of any dataset, generating an objective, quantitative score for the ‘FAIRness’ of that resource, together with advice on what caused any failures (https://www.nature.com/articles/s41597-019-0184-5). With this information, a data owner would be able to create a strategy to improve their FAIRness by focusing on “priority failures”. The public version of The FAIR Evaluator (https://w3id.org/AmIFAIR) has been used to assess >5500 datasets. Within the domain of rare disease registries, a recent publication about the VASCA registry shows how the Evaluator was used to track their progress towards FAIRness (https://www.medrxiv.org/content/10.1101/2021.03.04.21250752v1.full.pdf). To date, no resource – public or private – has ever passed all 22 tests, showing that FAIR assessment is able to provide guidance to even highly-FAIR resources.
The FAIR evaluation results can serve as a pointer to where your FAIRness can be improved.
[Hannah; this tool is included in the section above already: semi automated tools]
FAIR Guidance [https://www.ejprarediseases.org/fair_guidance/]
FAIR Assessment Tools
There is growing interest in the degree to which digital resources adhere to the goals of FAIR – that is, to be Findable, Accessible, Interoperable, and Retrievable by both humans and, more importantly, by machines acting on behalf of their human operator. Unfortunately, the path to FAIRness was left undefined by the original FAIR Principles paper, which chose to remain agnostic about which technologies or approaches were appropriate. As such, until recently, it has been impossible to make objectively valid statements about the degree to which a data object exhibits “FAIRness”.
With the encouragement of journal editors and other stakeholders who have a need to evaluate author/researcher claims regarding the FAIRness of their outputs, a group consisting of FAIR experts, journal editors, data repository hosts, internet researchers, and software developers assembled to jointly define a set of formal metrics that could be applied to test the FAIRness of a resource. The first edition of these metrics was aimed at self-assessment, in the form of a questionnaire; however, upon review of the validity of several completed self-assessments by data owners, we determined that the questions were often answered inconsistently, or incorrectly (knowingly or unknowingly), and often the data provider did not know enough about the data publishing environment to answer the questions at all. As such, a smaller group of FAIR experts created a second generation of FAIR Metrics that aimed to be fully automatable. The result was a set of 22 Metrics spanning most FAIR principles and sub-principles, which explicitly describe what is being tested, which FAIR Principle it applies to, why it is important to test this (meta)data feature, exactly how the test will be conducted, and what will be considered a successful result.
As a task under the objectives of the EJP RD, we created a set of software packages – The FAIR Evaluator – that coded each Metric into an automatable software-based test, and created an engine that could automatically apply these tests to any dataset, generating an objective, quantitative score for the ‘FAIRness’ of that dataset, together with advice on what caused any failures (https://www.nature.com/articles/s41597-019-0184-5). With this information, a data owner would be able to create a strategy to improve their FAIRness by focusing on “priority failures”. The public version of The FAIR Evaluator (https://w3id.org/AmIFAIR) has been used to assess >5500 datasets. Within the domain of rare disease registries, a recent publication about the VASCA registry shows how the Evaluator was used to track their progress towards fairness (https://www.medrxiv.org/content/10.1101/2021.03.04.21250752v1.full.pdf). To date, no resource – public or private – has ever passed all 22 tests, showing that FAIR assessment is able to provide guidance to even highly-FAIR resourcesGuidance [https://www.ejprarediseases.org/fair_guidance/]
FAIR Assessment Tools
There is growing interest in the degree to which digital resources adhere to the goals of FAIR – that is, to be Findable, Accessible, Interoperable, and Retrievable by both humans and, more importantly, by machines acting on behalf of their human operator. Unfortunately, the path to FAIRness was left undefined by the original FAIR Principles paper, which chose to remain agnostic about which technologies or approaches were appropriate. As such, until recently, it has been impossible to make objectively valid statements about the degree to which a data object exhibits “FAIRness”.
With the encouragement of journal editors and other stakeholders who have a need to evaluate author/researcher claims regarding the FAIRness of their outputs, a group consisting of FAIR experts, journal editors, data repository hosts, internet researchers, and software developers assembled to jointly define a set of formal metrics that could be applied to test the FAIRness of a resource. The first edition of these metrics was aimed at self-assessment, in the form of a questionnaire; however, upon review of the validity of several completed self-assessments by data owners, we determined that the questions were often answered inconsistently, or incorrectly (knowingly or unknowingly), and often the data provider did not know enough about the data publishing environment to answer the questions at all. As such, a smaller group of FAIR experts created a second generation of FAIR Metrics that aimed to be fully automatable. The result was a set of 22 Metrics spanning most FAIR principles and sub-principles, which explicitly describe what is being tested, which FAIR Principle it applies to, why it is important to test this (meta)data feature, exactly how the test will be conducted, and what will be considered a successful result.
Generic
If driving user question(s) were defined in Step 1 it should be “answered” in this step. The results of these question(s) are gathered by processing the FAIR machine-readable data. If RDF is the machine-readable format used, then RDF data stores (triple stores) are used to store the machine-readable data, and SPARQL queries are used to retrieve the data required to answer the driving user question(s).
...