Page Comparison

...

Short description

’FAIR evaluation results can serve as a pointer to where your FAIRness can be improved.’ (FAIRopoly)

In this pre-FAIRification phase you assess whether your (meta)data already contains meets FAIR featurescriteria, such as persistent unique identifiers for data elements and rich metadata, by . By using FAIRness assessment tooling [Generic]. By quantifying you can quantify the level of FAIRness of the data based on its current characteristics and environment, the . The assessment outcomes can help shape the necessary steps and requirements needed to achieve the desired FAIRification objectives [FAIRInAction]. The outcomes of this assessment can be used to compare against in the Assess FAIRness step (step X) to track the progress of you data towards FAIRness.[Jolanda] Different assessment tools are available which are (see A Generic Workflow for the Data FAIRification Process and FAIR in Action Framework by FAIRplus).

The how-to section describes a variety of assessment tools based on the FAIR principles.    

Why is this step important 

This still step will help you assess the current FAIRness level of your data, which can help . Comparing the current FAIRness to the previously defined FAIRification objectives will help you shape the necessary steps and requirements needed to achieve your FAIRification goals [Jolanda] (see this step)[Jolanda].  

Expertise requirements for this step 

This section could describe the expertise required. Perhaps the Build Your Team step could then be an aggregation of all the “Expertise requirements for this step” steps that someone needs to fulfil his/her FAIRification goals.  

How to 

[Sander]

FAIRCookbook

Assessment Chapter in the FAIRCookbook. It currently has recipes for two tools (no idea how they work yet):

1. FAIR Evaluator

2. FAIRshake

and help you create your solution plan. Furthermore, the assessment can be repeated in the Assess FAIRness step, allowing you to compare the results and check the progress of your data towards FAIRness.

How to 

Step 1

There are many tools that help to assess the FAIRness of your (meta)data before starting the FAIRification process:

• for an overview of available tools see FAIRassist;

• several tools are evaluated and compared in FAIR assessment tools: evaluating use and performance

...

• reviews ten FAIR assessment tools that have been evaluated and characterized using two datasets from the nanomaterials and microplastics risk assessment domain.

• we evaluated FAIR assessment tools in terms of 1) the prerequisite knowledge needed to run the tools, 2) the ease and effort needed to use them and 3) the output of the tool, with respect to the information it contains and the consistency between tools. This should help users, e.g., in the nanosafety domain, to improve their methods on storing, publishing and providing research data. To do this we provide guidance for researchers to pick a tool for their needs and be aware of its strong points and weaknesses.

• The selected tools were split up into four different sections, namely online self-assessment/survey, (semi-)automated, offline self-assessment and other types of tools. The tool selection was based on online searches in June 2020.

• They compare:

  • Online self-assessment survey

    • FAIRdat (I think development stopped on this one)

    • FAIRenough? (was replaced by FAIRaware)

    • ARDC FAIR self assessment

    • FAIRshake

    • SATIFYD

  • Online (Semi-) automated

    • Ammar, A. et al. and this one

    • FAIR evaluator software

  • Offline self-assessment

    • RDA-SHARC Simple Grids

    • GARDIAN (link from paper is dead, may be somewhere around here, can’t find it though)

  • Other

    • Data Stewardship Wizard

There is also FAIRassist with lots of tools and questionnaires:

• These resources include manual questionnaires, checklists and automated tests that help users understand how to achieve a state of "FAIRness", and how this can be measured and improved.

More Checklists and tools:

• A Checklist produced for use at the EUDAT summer school to discuss how FAIR the participant's research data were and what measures could be taken to improve FAIRness:

  • https://zenodo.org/records/1065991#.Xs_XpC2cbOQ%C2%A0

[Hannah] There are also these tools [Mijke: these are the ones Nivel used in a recent project - have them write the community example?]:

FIP Mini Questionnaire from GO-FAIR: https://www.go-fair.org/how-to-go-fair/fair-implementation-profile/fip-mini-questionnaire/

Data Maturity Model: https://zenodo.org/records/3909563#.YGRNnq8za70

[Sander]

Hannah mentions the Data Maturity Model. This is also here on FAIRplus. There is also this Github from FAIRplus and the sheet for the actual assessment is here. Could be worrying: last update was last year.

Related: in the FAIRtoolkit they describe Data Capability Maturity Model:

• Most recently, CMM has been adapted by the FAIRplus IMI consortium [7] to improve an organisation’s life science data management process, which is the basis for the method described here.

• The FAIR data CMM method identifies 1) important organisational aspects of FAIR data transformation and management, 2) a sequence of levels that form a desired path from an initial state to maturity and 3) a set of maturity indicators for measuring the maturation levels.

  • e.g. Findability Maturity Indicators. Also describes some team requirements.

Furthermore: FAIR Evaluator (FAIRopoly and FAIR Guidance) – text copied below. 

FAIRopoly 

• As a task under the objectives of the EJP RD, we created a set of software packages – The FAIR Evaluator – that coded each Metric into an automatable software-based test, and created an engine that could automatically apply these tests to the metadata of any dataset, generating an objective, quantitative score for the ‘FAIRness’ of that resource, together with advice on what caused any failures (https://www.nature.com/articles/s41597-019-0184-5).  With this information, a data owner would be able to create a strategy to improve their FAIRness by focusing on “priority failures”.  The public version of The FAIR Evaluator (https://w3id.org/AmIFAIR) has been used to assess >5500 datasets.  Within the domain of rare disease registries, a recent publication about the VASCA registry shows how the Evaluator was used to track their progress towards FAIRness (https://www.medrxiv.org/content/10.1101/2021.03.04.21250752v1.full.pdf). To date, no resource – public or private – has ever passed all 22 tests, showing that FAIR assessment is able to provide guidance to even highly-FAIR resources. 

• The FAIR evaluation results can serve as a pointer to where your FAIRness can be improved. 

FAIR Guidance [https://www.ejprarediseases.org/fair_guidance/] 

FAIR Assessment Tools 

There is growing interest in the degree to which digital resources adhere to the goals of FAIR – that is, to be Findable, Accessible, Interoperable, and Retrievable by both humans and, more importantly, by machines acting on behalf of their human operator. Unfortunately, the path to FAIRness was left undefined by the original FAIR Principles paper, which chose to remain agnostic about which technologies or approaches were appropriate. As such, until recently, it has been impossible to make objectively valid statements about the degree to which a data object exhibits “FAIRness”.   

With the encouragement of journal editors and other stakeholders who have a need to evaluate author/researcher claims regarding the FAIRness of their outputs, a group consisting of FAIR experts, journal editors, data repository hosts, internet researchers, and software developers assembled to jointly define a set of formal metrics that could be applied to test the FAIRness of a resource.  The first edition of these metrics was aimed at self-assessment, in the form of a questionnaire; however, upon review of the validity of several completed self-assessments by data owners, we determined that the questions were often answered inconsistently, or incorrectly (knowingly or unknowingly), and often the data provider did not know enough about the data publishing environment to answer the questions at all.  As such, a smaller group of FAIR experts created a second generation of FAIR Metrics that aimed to be fully automatable.  The result was a set of 22 Metrics spanning most FAIR principles and sub-principles, which explicitly describe what is being tested, which FAIR Principle it applies to, why it is important to test this (meta)data feature, exactly how the test will be conducted, and what will be considered a successful result.  

As a task under the objectives of the EJP RD, we created a set of software packages – The FAIR Evaluator – that coded each Metric into an automatable software-based test, and created an engine that could automatically apply these tests to any dataset, generating an objective, quantitative score for the ‘FAIRness’ of that dataset, together with advice on what caused any failures (https://www.nature.com/articles/s41597-019-0184-5).  With this information, a data owner would be able to create a strategy to improve their FAIRness by focusing on “priority failures”.  The public version of The FAIR Evaluator (https://w3id.org/AmIFAIR) has been used to assess >5500 datasets.  Within the domain of rare disease registries, a recent publication about the VASCA registry shows how the Evaluator was used to track their progress towards fairness (https://www.medrxiv.org/content/10.1101/2021.03.04.21250752v1.full.pdf).  To date, no resource – public or private – has ever passed all 22 tests, showing that FAIR assessment is able to provide guidance to even highly-FAIR resources.  

Generic 

If driving user question(s) were defined in Step 1 it should be “answered” in this step. The results of these question(s) are gathered by processing the FAIR machine-readable data. If RDF is the machine-readable format used, then RDF data stores (triple stores) are used to store the machine-readable data, and SPARQL queries are used to retrieve the data required to answer the driving user question(s). 

FAIRCOOKBOOK recipe: [https://faircookbook.elixir-europe.org/content/recipes/introduction/fairification-process.html] 

Phase 3: assess, design, implement, repeat 

Following the selection of the “action” team, an iterative cycle of assessment, design, and implementation in put in place. 

Assessment : Prior to starting the work, the assessment of goals is done to ensure that individuals in the action team are updated and clear with the FAIRification goals formulated by the data owners. This assessment is carried out by review team which could be an independent team or certain individuals from the technical team who are not involved in the action team. The assessment involves a binary decision of “GO” or “NO GO” based on the FAIRification goals and the catalog provided. At this stage, the reviews can also provide suggestion based on their experiences on the resources, tool, or goals. 

Design : Once the team receives a “GO” decision from the review team, the action team now starts by enlisting the steps that need to be done performed to achieve the goal. For each task, the resources, an estimate time duration, as well as the responsible person is selected. 

Implementation : Once the tasks have been selected and assigned, the actual work begins. To ensure that the action team is working smoothly, weekly or bi-weekly meetings is recommended so that the team is aware of the progress. 

Once the implementation of task listed in the design phase are done, the action team assess the work done and checks the aligned with the FAIRification goal. In case more tasks are needed to achieve the goal, a second round of the assess-review-implement cycle takes place as described above with the starting point as the FAIRification goals, the completed tasks and the proposed task 

This phase is usually run in short sprints of 3-month. 

Practical Examples from the Community 

This section should show the step applied in a real project. Links to demonstrator projects. 

[Mijke: Nivel has done a pre-assessment in a recent project - have them write the community example? The ZonMw program have written FAIR Improvement Plans, we can contact some of those and ask for example]

References & Further reading

[FAIRopoly] https://www.ejprarediseases.org/fairopoly/  

[FAIRinAction] https://www.nature.com/articles/s41597-023-02167-2 

[Generic] https://direct.mit.edu/dint/article/2/1-2/56/9988/A-Generic-Workflow-for-the-Data-FAIRification   

Authors / Contributors 

...

• ;

• RDMkit discusses several solutions.

While we focus specifically on the FAIRness of (meta)data in this step, it is also possible to possible to assess general FAIR awareness, for example by using the FAIR Aware tool provided by DANS.

Step 2

Decide which tool fits your goal(s) the best. Broadly, the tools fall into the two categories described below.

• Online self-assessment surveys. Here, the user is presented with an online form, which is filled in manually.

• (Semi) automated tests. Here (semi) automated tests are performed on a dataset by providing the tool with, for example, a link to an already published dataset.

In both cases, the result gives an indication about the FAIRness of the (meta)data. Additionally, tools may give advice how to improve FAIRness. It is important to bear in mind that outcomes of tools may vary due to, for example, differences in tests performed and subjectivity of self-assessments surveys. See EOSC’s FAIR Assessment Tools: Towards an “Apples to Apples” Comparisons for more information this.

The tables below provide an overview of some of the more popular tools from both categories.

Online self-assessment surveys

Online (Semi-) automated tests

FAIR Implementation profiles

Another promising development is the FAIR Implementation Profile (FIP), developed by the GO FAIR Foundation. Once published, a FIP can be reused by others, thus acting as a recipe for making data FAIR by a community, for example a research project or an institute, based on agreements and standards within that community. A FIP can be used to compare your currently used FAIR implementation choices, such as standards used in your dataset, to those used by your community, thus providing a Pre-FAIR score. FIPs and their usage are currently still under active development. For more information, see Creating a FAIR Implementation Profile (FIP), FIP Mini Questionnaire and the FIP Data Stewardship Wizard.

Step 3

To successfully do a pre-FAIR assessment, do the following:

• learn from examples (see the practical examples section);

• familiarise yourself with the tool you intend to use;

• involve the necessary experts (see expertise requirements section);

• perform the assessment.

The final evaluation will give insight into the current FAIRness of your data. Depending on the tool used, you may receive feedback on how to improve the FAIRness of your data. Thus, the outcome of the pre-FAIR assessment helps you determine the next steps to achieve your FAIRification goals.

Expertise requirements for this step 

The expertise required may depend on the assessment tool you want to use. Experts that may need to be involved, as described in Metroline Step: Build the Team, are described below.

• FAIR data stewards. Specialist who can help filling out the surveys and questionnaires.

• Research software engineers. Specialists whocan help running some of the specialised software.

Practical examples from the community 

Pending - Nivel Example

Training

Relevant training will be added in the future if available.

Suggestions

Visit our How to contribute page for information on how to get in touch if you have any suggestions about this page.