Documentation & data schemas for the externally available bulk data releases


The Dimensions bulk export is designed for users who need to do analysis over significant portions of the Dimensions data. This document is intended as a quick overview of how you can use this and what the data looks like.

While Dimensions provides an API for easily accessing specific documents or groups of documents with various filters, some tasks require using a significant portion of the total data. There are two key difficulties:

  1. Accessing millions of individual documents through the API can take time
  2. Keeping up to date, identifying which documents to update

The Dimensions bulk export is aimed at tackling these two issues. It is not intended as a replacement for the API, but as a complement. Many tasks will be best served by the API, and we would highly encourage users to start with the API rather than rebuilding what we can offer. However, if you are doing analysis over a significant portion of the data (network analysis is a common use case), then the bulk export should make things easier.

General information on the data

Overall structure

The data is provided as folders on AWS S3. The data is put together as JSONL files, where each line is a single JSON document representing a publication, grant, patent document, etc.. For the larger datasets the file will contain up to 10000 documents (typically 10000 but sometimes smaller). These files should not be relied on to contain any logical grouping (e.g. by date or publisher), the batch sizes are chosen to make downloading/processing simpler and specific sizes should not be relied upon. More specific information on the regularity and the folder structure can be found in the dedicated document on the different document types.

Delivery Overview

The following table shows the regularity and the amount of data to be expected as of January 2022. Updates are normally delivered as a baseset and consecutive _update_s. If a particular consecutive update is missing, it means there are no updates available for that day.

  Delivery Amount of records Size regularity
Categories always full set ~1000 <100 KB On demand - when we add new categories
Organisations always full set ~100 K <130MB 3-4 times per year
Publications baseset ~130 M > 3 TB 2-4 times a year
  update 8 - 80 K About 20 - 800 MB daily
Patents baseset ~150 M About 360 GB 2-4 times a year
  updates 50K - 1M About 4 - 30 GB weekly
Grants always full set 6 M About 14 GB 10-12 times a year
Clinical Trials baseset 700 K About 2.5 GB 2-4 times a year
  updates 1 - 5K 5 MB - 35 MB daily
Data sets baseset 12 M About 22 GB 1-2 times a year
  updates 2 - 200 K 5 MB - 2 GB daily
Policy Documents baseset 740 K ~550 MB 1-3 times a year
  updates 1 - 10 K 10 MB weekly
(Technical) Reports baseset About 1 M About 15 GB 1-2 times a year
  updates 1 - 10 K 5 - 25 MB daily

Secure Data Access

For accessing the Dimensions data, a specific folder in Amazon S3 is provided where the data can be accessed in a read only format. The credentials will be passed along using secure communication.

Dimensions provides AWS credentials to allow access for a variety of tools, e.g.:

Configuring your tool with credentials:

S3 Bucket Path:	ai.dimensions.data/sourcename
Access Key ID:      XXXX
Secret Access Key:  XXXX

Make sure that you put in the whole path, including the source you want to access. You will find the specific path in the documentation of each source further down below in the “S3 bucket path” chapters. After setting up credentials in the chosen tool, downloading the file is a simple drag and drop operation.

To automate access you can also use Amazon’s command line tools. Once you have access to a certain sourcename, you can use the aws s3 ls command to get a list of all the releases available:

aws s3 ls s3://ai.dimensions.data/sourcename/

A single release is always identified by an additional date YYYYMMDD as part of the path marking the release. With Amazon’s command line tools you can now download a single release:

aws s3 cp s3://ai.dimensions.data/sourcename/YYYYMMDD target

To download significant amounts or to pull all updates, the sync command is very helpful (it only downloads new or not successfully downloaded files):

aws s3 sync s3://ai.dimensions.data/sourcename/YYYYMMDD target

Note that for both cp and sync the target can be another location on S3.

If you store your data there, it can be much faster to copy things directly like this rather than via your own machines. This is especially true within the same region (US-EAST-1) if you set your threads high.

Data delivery

Most of the data will be provided as an initial, large set of files called a baseset. The initial folder will look something like this:


Grants will be provided as a single folder with each release. The other releases will receive regular updates during a release.

For new data, a new folder will be created with new and updated documents on a regular basis. These documents will be complete, that is if anything changes then the whole document will appear again (so no complex diffing is required):


The folders will be orderable lexicographically, and there is no correspondence between the filenames in one update to another. 0000000099_0000000099/records_0000001.jsonl does not contain the updates to 0000000001_0000000098/records_0000001.jsonl it is simply the first batch of updated files.

While these may seem like restrictions, the goal is to provide you with a simple process for getting up to date quickly rather than provide complex guarantees for random access.

Download each folder in order (lexicographically) If an item has an ID you have seen before, the previous document with that ID should be replaced.

When items are deleted (for example, when they are discovered to be a duplicate and merged) they will appear with their ID and an obsolete status (details in the schemas).

New basesets

Over time, significant updates will happen across the data. To prevent the situation of having to process years of daily updates and seeing the same documents repeatedly, a new baseset will be created. The current and previous baseset will always be updated, to smooth the switchover. This allows time to prepare for any format changes as well.

Each provided group of baseset & updates will be placed in a folder with a date, and so you can pick the latest date for the latest data.

Data formats


There are several structures in the data that are shared between different content types. The current latest formats for these are

People for publication authors, grant investigators, etc.

Funding for links to grants and funders.

Categories for classification into FOR, RCDC, etc.


The categories contain IDs that can be looked up in the categories lookup releases in the following location


Over time, we update the machine learning models or training data behind the classifications. When there are significant changes, we update the version of the model. The version used for a classification is provided in the categories list under the key version.

To support migration from one model to another, new models will be added in preview before being released to production. Once a new model is released, the previous version will be deprecated.

Currently, all production models are version 1.

Category type Version Status
broad_research_areas 1 old
broad_research_areas 2020 production
cancer_types 1 old
cancer_types 2020 production
cso 1 old
cso 2020 production
for 1 production
for 2020 preview
ford 2020 preview
health_research_areas 1 production
health_research_areas 2020 preview
hrcs_hc 1 old
hrcs_hc 2020 production
hrcs_rac 1 old
hrcs_rac 2020 production
rcdc 1 production
rcdc 2020 preview
sdg 1 production
sdg 2021 preview
uoa 1 production
uoa 2020 preview
for_2020 2022 preview
arxiv_cs 2022 preview


S3 bucket path

The S3 bucket path to access organisations: s3://ai.dimensions.data/organisations/

Data format

The organisations data is delivered as one JSONL file 3-4 times a year. No updates are provide between those releases. The delivery contains documents that may be active (see JSON schema here), obsolete (see JSON schema here) or redirected (see JSON schema here).


S3 bucket path

The S3 bucket path to access publications:


Data format

With the latest release in s3://ai.dimensions.data/publications/20220105/ we started providing the data as compressed gzip files. The release contains documents that may be active (see version 8.schema.json here) or obsolete (see JSON schema here).

Version 8 release notes

This format version is backwards compatible with versions 5, 6 and 7, no fields have been removed.

There are 2 new fields to be aware of:

Also the categories list now provides a new preview version 2021 of sdg and a preview version 2020 of ford.

Version 7 release notes

This format version is backwards compatible with versions 5 and 6, no fields have been removed.

There are several new fields to be aware of:


S3 bucket path

The S3 bucket path to access patents:


Data format

The latest release in s3://ai.dimensions.data/patents/20210309/ contains documents that may be active or obsolete in format version 3 (see JSON schema here). active can be either updates or new documents, while obsolete indicates that the previous version of this documentshould be deleted.


S3 bucket path

The S3 bucket path:


Data format

With the release in s3://ai.dimensions.data/grants/20201020/ we started provding documents that comply with this format: see JSON schema here.

Clinical Trials

S3 bucket path

The S3 bucket path:


Data format

With the release in s3://ai.dimensions.data/clinicaltrials/20210615/ we started provding documents that comply with this format: see JSON schema here.

Data sets

S3 bucket path

The S3 bucket path:


Data format

With the release in s3://ai.dimensions.data/datasets/20210929/ we started providing the data as compressed gzip files. The records comply with this format: see JSON schema here.

Policy Documents

S3 bucket path

The S3 bucket path to access policy documents:


Data format

The latest release in s3://ai.dimensions.data/policydocuments/20210428 contains documents in this format: see JSON schema here.

(Technical) Reports

The S3 bucket path to access reports:


Data format

Documents provided validate against this JSON schema.