Page Comparison

Short Description Description

In this pre-FAIRification phase you assess whether your (meta)data already contains FAIR features, such as persistent unique identifiers for data elements and rich metadata, by using FAIRness assessment tooling [Generic].

...

Expertise requirements for this step 

This section could describe the The expertise required . Perhaps the Build Your Team step could then be an aggregation of all the “Expertise requirements for this step” steps that someone needs to fulfil his/her FAIRification goals.  [Hannah: I would say expertise depends a bit on which tool you use; most checklists and questionnaires are pretty low effort and self-explanatory. But some of the more automated tools require some (programming) skills]may depend on the assessment tool you want to use. Experts that may need to be involved, as described in Metroline Step: Build the Team, include:

• Data stewards: can help filling out the surveys and questionnaires

• Research Software Engineer: can help running some of the specialised software

• ELSI experts: can help filling out the ELSI related questions surveys and questionnaires

How to 

There are many assessment tools to do a pre-FAIR assessment of your (meta)data. FAIRassist holds a (manually created) collection with various different tools. These include manual questionnaires or checklists and automated tests that help users understand how to achieve a state of "FAIRness", and how this can be measured and improved. Furthermore, a 2022 publication (FAIR assessment tools: evaluating use and performance) compared a number of tools. Of these and the tools listed on FAIRassist, we suggest that the following can be considered for your pre-FAIR assessment:

...

These tools allow you to fill in an online form. The result of the survey can be e.g. a score to indicate the FAIRness of your (meta)data. Some tools additionally provide advice on how to improve FAIRness at different levels.. Well-known online surveys include:

Online (Semi-) automated

These tools do an automatic assessment by reading the metadata available at a certain URI.

• Ammar, A. et al. [Hannah: this links to another page on the confluence] and this one [Hannah; these are jupyter notebooks to use for data from specific databases; can be extended/adjusted with your own dataset; it seems a bit of a larger effort to use for a ‘quick’ FAIR assessment of your (meta)data]

• FAIR evaluator software

• FAIRchecker; this tool automatically provides a score for all aspects of FAIR from a URI

Offline self-assessment

• RDA-SHARC Simple Grids

• GARDIAN (link from paper is dead, could be somewhere around here, can’t find it though)

...

Online (Semi-) automated

For even more surveys and (semi-) automated tools, see FAIRassist.

The FAIR Data Maturity Model

FAIR assessment tools vary greatly in their outcomes. The FAIR Data Maturity Model (created by the Research Data Alliance, or RDA) aims to harmonise outcomes of FAIR assessment tools to make these comparable. Based on the FAIR principles and sub-principles, they have created a list of universal 'maturity indicators'. Their work resulted in a checklist (with extensive description of al maturity indicators), which can be used to assess the FAIRness of your (meta)data.

FAIR maturity evaluation system

FAIR Implementation Profiles (FIPs)

Potentially: compare the community FIP with your own fingerprint. This gives an indication on whether you meet R1.3?

‘The FAIR Principle R1.3 states that “(Meta)data meet domain-relevant Community standards”. This is the only explicit reference in the FAIR Principles to the role played by domain-specific communities in FAIR. It is interesting to note that an advanced, online, automated, FAIR maturity evaluation system [22] did not attempt to implement a maturity indicator for FAIR Principle R1.3. It was not obvious during the development of the evaluator system how to test for “domain-relevant Community standards” as there exists, in general, no venue where communities publicly and in machine-readable formats declare data and metadata standards, and other FAIR practices. We propose the existence of a valid, machine-actionable FIP be adopted as a maturity indicator for FAIR Principle R1.3.’

[Hannah] There are also these tools [Mijke: these are the ones Nivel used in a recent project - have them write the community example?]:

FIP Mini Questionnaire from GO-FAIR: https://www.go-fair.org/how-to-go-fair/fair-implementation-profile/fip-mini-questionnaire/

Data Maturity Model: https://zenodo.org/records/3909563#.YGRNnq8za70

[Mijke: RDMkit has a page on this → https://rdmkit.elixir-europe.org/compliance_monitoring#how-can-you-measure-and-document-data-management-capabilities ] 

[Sander]

FAIRCookbook

Assessment Chapter in the FAIRCookbook. It currently has recipes for two tools (no idea how they work yet):

1. FAIR Evaluator

2. FAIRshake

FAIR assessment tools: evaluating use and performance paper (2022):

• reviews ten FAIR assessment tools that have been evaluated and characterized using two datasets from the nanomaterials and microplastics risk assessment domain.

• we evaluated FAIR assessment tools in terms of 1) the prerequisite knowledge needed to run the tools, 2) the ease and effort needed to use them and 3) the output of the tool, with respect to the information it contains and the consistency between tools. This should help users, e.g., in the nanosafety domain, to improve their methods on storing, publishing and providing research data. To do this we provide guidance for researchers to pick a tool for their needs and be aware of its strong points and weaknesses.

• The selected tools were split up into four different sections, namely online self-assessment/survey, (semi-)automated, offline self-assessment and other types of tools. The tool selection was based on online searches in June 2020.

• They compare:

  • Online self-assessment survey

    • FAIRdat (I think development stopped on this one); you get stars (1 to 5) for the letters of FAIR, but cannot safe the results of the assessment

    • FAIRenough? (was replaced by FAIRaware)

    • ARDC FAIR self assessment

    • FAIRshake

    • SATIFYD

  • Online (Semi-) automated

    • Ammar, A. et al. and this one

    • FAIR evaluator software

  • Offline self-assessment

    • RDA-SHARC Simple Grids

    • GARDIAN (link from paper is dead, may be somewhere around here, can’t find it though)

  • Other

    • Data Stewardship Wizard

More Checklists and tools:

• A Checklist produced for use at the EUDAT summer school to discuss how FAIR the participant's research data were and what measures could be taken to improve FAIRness:

  • https://zenodo.org/records/1065991#.Xs_XpC2cbOQ%C2%A0 [Hannah; this is also an offline checklist; not sure if we should recommend to consider. I also think it is rather limited compared to the rest of the tools/checklists]

[Sander]

Hannah mentions the Data Maturity Model. This is also here on FAIRplus. There is also this Github from FAIRplus and the sheet for the actual assessment is here. Could be worrying: last update was last year.

[Hannah: I cannot really find a clear description on how to use it, only a huge excel file (for which you have to dig quite deeply into the GitHub, maybe we can link to it here if we include it https://github.com/FAIRplus/Data-Maturity/tree/master/docs/assessment ?]

Related: in the FAIRtoolkit they describe Data Capability Maturity Model:

• Most recently, CMM has been adapted by the FAIRplus IMI consortium [7] to improve an organisation’s life science data management process, which is the basis for the method described here.

• The FAIR data CMM method identifies 1) important organisational aspects of FAIR data transformation and management, 2) a sequence of levels that form a desired path from an initial state to maturity and 3) a set of maturity indicators for measuring the maturation levels.

  • e.g. Findability Maturity Indicators. Also describes some team requirements.

[Hannah; I think this is also more about assessing FAIR in an organization?The FAIRplus dataset maturity indicators were created based on previous work around the FAIR indicators, done by the Research Data Alliance (RDA) and the FAIRsFAIR projects:

• FAIR Data Maturity Model Working Group. (2020). FAIR Data Maturity Model. Specification and Guidelines (1.0). https://doi.org/10.15497/rda00050

• Devaraju, Anusuriya, Huber, Robert, Mokrane, Mustapha, Herterich, Patricia, Cepinskas, Linas, de Vries, Jerry, L’Hours, Herve, Davidson, Joy, & Angus White. (2020). FAIRsFAIR Data Object Assessment Metrics (0.4). Zenodo. https://doi.org/10.5281/zenodo.4081213

In the definitions of the FAIRplus-DSM indicators, you will find a link to the corresponding RDA or FAIRsFAIR indicator when they are related.

Hannah mentions the Data Maturity Model. This is also here on FAIRplus. There is also this Github from FAIRplus and the sheet for the actual assessment is here. Could be worrying: last update was last year.

[Hannah: I cannot really find a clear description on how to use it, only a huge excel file (for which you have to dig quite deeply into the GitHub, maybe we can link to it here if we include it https://github.com/FAIRplus/Data-Maturity/tree/master/docs/assessment ?]

Related: in the FAIRtoolkit they describe Data Capability Maturity Model:

• Most recently, CMM has been adapted by the FAIRplus IMI consortium [7] to improve an organisation’s life science data management process, which is the basis for the method described here.

• The FAIR data CMM method identifies 1) important organisational aspects of FAIR data transformation and management, 2) a sequence of levels that form a desired path from an initial state to maturity and 3) a set of maturity indicators for measuring the maturation levels.

  • e.g. Findability Maturity Indicators. Also describes some team requirements.

[Hannah; I think this is also more about assessing FAIR in an organization?]

FAIR maturity evaluation system

FAIR Implementation Profiles (FIPs)

Potentially: compare the community FIP with your own fingerprint. This gives an indication on whether you meet R1.3?

‘The FAIR Principle R1.3 states that “(Meta)data meet domain-relevant Community standards”. This is the only explicit reference in the FAIR Principles to the role played by domain-specific communities in FAIR. It is interesting to note that an advanced, online, automated, FAIR maturity evaluation system [22] did not attempt to implement a maturity indicator for FAIR Principle R1.3. It was not obvious during the development of the evaluator system how to test for “domain-relevant Community standards” as there exists, in general, no venue where communities publicly and in machine-readable formats declare data and metadata standards, and other FAIR practices. We propose the existence of a valid, machine-actionable FIP be adopted as a maturity indicator for FAIR Principle R1.3.’

[Hannah] There are also these tools [Mijke: these are the ones Nivel used in a recent project - have them write the community example?]:

FIP Mini Questionnaire from GO-FAIR: https://www.go-fair.org/how-to-go-fair/fair-implementation-profile/fip-mini-questionnaire/

Data Maturity Model: https://zenodo.org/records/3909563#.YGRNnq8za70

[Mijke: RDMkit has a page on this → https://rdmkit.elixir-europe.org/compliance_monitoring#how-can-you-measure-and-document-data-management-capabilities ] 

[Sander]

FAIRCookbook

Assessment Chapter in the FAIRCookbook. It currently has recipes for two tools (no idea how they work yet):

1. FAIR Evaluator

2. FAIRshake

More Checklists and tools:

• A Checklist produced for use at the EUDAT summer school to discuss how FAIR the participant's research data were and what measures could be taken to improve FAIRness:

  • https://zenodo.org/records/1065991#.Xs_XpC2cbOQ%C2%A0 [Hannah; this is also an offline checklist; not sure if we should recommend to consider. I also think it is rather limited compared to the rest of the tools/checklists]

Furthermore: FAIR Evaluator (FAIRopoly and FAIR Guidance) – text copied below. 

...