Schema2QA 1.0

Overview

The page lists statistics and results of the original Schema2QA dataset published on Schema2QA paper

Statistics

Train:

	restaurants	people	movies	books	music	hotels	average
# properties supported	25	13	16	15	19	18	17.7
Synthetic	270,081	270,081	270,081	270,081	270,081	270,081	270,081
Human Paraphrase	6,419	7,108	3,774	3,941	3,626	3,311	4,697
Total (after augmentation)	508,101	614,841	405,241	410,141	425,041	377,341	456,784

Evaluation:

	restaurants	people	movies	books	music	hotels	average
Validation	528	499	389	362	326	443	424.5
Test	524	500	413	410	288	528	443.8

Leader board

	restaurants	people	movies	books	music	hotels	average
Schema2QA	69.7%	75.2%	70.0%	70.0%	63.9%	67.0%	69.3%
AutoQA (w/o human data)	65.3%	64.6%	66.1%	54.1%	57.3%	70.1%	62.9%

Download links

Validation data can be found under directories of each domain in v1.0. The full training data can be found on OVAL Wiki.

License

The dataset is released under CC BY 4.0. Please cite the following papers if use this dataset in your work:

% evaluation and human paraphrase data
@inproceedings{xu2020schema2qa,
  title={Schema2QA: High-Quality and Low-Cost Q\&A Agents for the Structured Web},
  author={Xu, Silei and Campagna, Giovanni and Li, Jian and Lam, Monica S},
  booktitle={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management},
  pages={1685--1694},
  year={2020}
}

% auto paraphrase data
@inproceedings{xu2020autoqa,
  title={AutoQA: From Databases to Q\&A Semantic Parsers with Only Synthetic Training Data},
  author={Xu, Silei and Semnani, Sina and Campagna, Giovanni and Lam, Monica},
  booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  pages={422--434},
  year={2020}
}