NeurIPS 2026 Evaluations & Datasets FAQ

NeurIPS 2026 Evaluations & Datasets FAQ This FAQ will be continually updated. Please bookmark this page and review it before submitting any questions. Note: Authors are also advised to consult the NeurIPS Main Track handbook , as general policies apply to ED submissions as well. General FAQs What is the LaTeX template for the ED track? It’s the same as the main track template. Check “Paper Formatting Instructions” at NeurIPS Main Track handbook . Are there guidelines for submissions which are from the 2024/2025 Competitions track, e.g., reporting on competition results? No, there are no special guidelines. Please follow the ED CFP and data hosting guidelines . Your submission will be reviewed according to the same standards alongside all other ED track submissions. We suggest you review the revised scope of the E&D track carefully when framing your work. Are dataset/code submissions due on May 6 (the full paper deadline)? Yes. We follow the Main Track timeline, so the full paper — including all required materials — must be submitted by May 6, 2026 (AOE). For the ED track, datasets and code are not considered supplementary materials. If your submission includes data and/or code, they must be submitted in their final form by May 6, 2026 (AOE), together with the full paper. What is the LaTeX configuration for a single-blind submission? Please use \usepackage[eandd, nonanonymous]{neurips_2026} if you wish to make your submission single-blind for the ED track. Is my paper a good fit for ED track? Please carefully read the ED CFP and use your best judgment. Track chairs cannot provide detailed advice on the relevance of your paper. The following questions might help further. My main contribution is a training dataset. Does it still fit the scope of E&D? Yes. Training datasets are welcome as long as the work clearly demonstrates their value in improving (downstream) evaluations, e.g., task performance, robustness, fairness, privacy, and alignment. The metric(s) and task(s) the dataset is designed to improve upon should be clearly stated, along with any assumptions and limitations. Submissions that propose a dataset with the “potential” for machine learning or task improvement without this demonstration are not in scope. How should I include code in my submission? Please see the “code guidelines” in the NeurIPS handbook . You will be asked to provide a URL to a hosting platform (e.g., GitHub, Bitbucket). If that is not an option, you can alternatively attach it as a ZIP file as supplementary material with your paper. All code should be documented and executable. My submission is a benchmark consisting of an environment for evaluation only/audits an existing benchmark using publicly available data/is a theoretical framework for comparing evaluation designs. Do I need to follow the data-hosting guidelines? No. If your submission does not introduce new data, you do not need to follow data-hosting guidelines. You do need to follow code-hosting guidelines if your submission includes new code or tools. The dataset-hosting and Croissant requirements apply only to submissions that introduce new datasets. Dataset hosting FAQs The Croissant format can’t handle the file type(s) in my dataset submission. What should I do? You should still submit a Croissant file. You can choose to provide only dataset-level metadata and a description of the resources in the dataset (FileObject and FileSet). You can omit RecordSets in this scenario. The recommended Croissant-compatible data hosting platforms should handle this gracefully for you, but you will need to address it manually if you decide to self-host your dataset. I have a submission consisting of multiple datasets. How do I submit the Croissant files? You should submit a Croissant file for every dataset (and check whether they are all valid). You can combine the .json files into a .zip folder and upload that during submission. In the dataset URL, refer to a webpage that documents the collection of datasets as a whole. The URLs for the individual datasets must be included in the Croissant files. How do we handle our submission which includes a private hold-out set which we wish to keep private and unreleased, e.g., to avoid potential contamination? You should mention that you have a private hold-out set and describe it in your paper, but the main contribution of your paper should be the publicly released portion of your dataset. The publicly released portion of your dataset needs to conform to the data hosting guidelines. It may also contain a public validation and test set collected using the same protocol as the private one. My submission includes a synthetic dataset. Does it need to be documented and hosted in the same way? Yes. All data hosting guidelines apply to synthetic datasets as well. I don’t want to make my dataset publicly accessible at the time of submission. What are my options? Both the Harvard Dataverse and Kaggle platforms offer private URL preview link sharing. This means your dataset is accessible only to those who have the special URL, e.g., reviewers. Note that you will be required to make your dataset public by the camera-ready deadline. Failure to do so may result in removal from the conference and proceedings. Can I make changes to my dataset after I have made my submission to Open Review? You can make changes until the submission deadline. After the submission deadline, we will perform automated verification checks of your dataset to assist in streamlining and standardizing reviews. If it changes in a way that invalidates the original reviews at any time between the submission deadline and by the camera ready deadline or publication of proceedings, we reserve the right to remove it from the conference or proceedings. I am experiencing problems with the platform I am using to release my dataset. What should I do? We have worked with maintainers of the dataset hosting platforms to identify the appropriate contact information for authors to use for support in case of issues or help with workarounds for storage quotas, etc. Find this contact information in the ED data hosting guidelines I need to require credentialized (AKA gated) access to my dataset This will be possible on the condition that a credentialization is necessary for the public good (e.g. because of ethically sensitive medical data), and that an established credentialization procedure is in place that is 1) open to a large section of the public; 2) provides rapid response and access to the data; and 3) is guaranteed to be maintained for many years. A good example here is PhysioNet Credentialing, where users must first understand how to handle data with human subjects, yet is open to anyone who has learned and agrees with the rules. This should be seen as an exceptional measure, and NOT as a way to limit access to data for other reasons (e.g., to shield data behind a Data Transfer Agreement). Misuse would be grounds for desk rejection. During submission, you can indicate that your dataset involves open credentialized access, in which case the necessity, openness, and efficiency of the credentialization process itself will also be checked. Our dataset requires credentialized access. How do we preserve single-blind review, i.e., ensure the identities of reviewers aren’t shared with authors? If it’s possible to share a private preview link rather than requiring credentials, you may try that. Or, you can make an account, give it view access to the dataset, and share login details with reviewers. After submission, you can send a private message visible only to reviewers on Open Review. I have an extremely large (> 1 TB) dataset. How do I allow reviewers to properly evaluate it? Please make sure that the full dataset is available at submission time. You can in addition provide ways to help reviewers explore your dataset. This could be a notebook that downloads a portion of the data and helps you explore it, a smaller data sample (ideally hosted in the same way), or a bespoke solution appropriate for your dataset. If you make a sample, also explain how you created that sample. Our submission involves using existing public datasets. Do we need to host these in accordance with the data hosting guidelines? No, but you should make any code used to modify or otherwise use the public datasets, e.g., for a new benchmark that you are submitting, accessible and executable (meaning you will need to provide publicly accessible links to the data sources used). You also should not claim the existing public datasets as part of your submission. The online app for checking the validity of croissant files runs for a long time and times out. This can happen when you have a dataset on Hugging Face. The app may be rate-limited, which causes an error and automatic restarts. If this happens, we recommend validating your Croissant file locally. You can click the three dots at the top right of the app to get code to run it locally, or clone the repository and run it in your own HF Space. Successful Page Load NeurIPS uses cookies for essential functions only. We do not sell your personal information. Our Privacy Policy » Accept