Welcome to the webpage for our EMNLP 2021 position paper: Evaluation Paradigms in Question Answering.

Abstract

Question answering (QA) primarily descends from two branches of research: (1) Alan Turing's investigation of machine intelligence at Manchester University and (2) Cyril Cleverdon's comparison of library card catalog indices at Cranfield University. This position paper names and distinguishes these paradigms. Despite substantial overlap, subtle but significant distinctions exert an outsize influence on research. While one evaluation paradigm values creating more intelligent QA systems, the other paradigm values building QA systems that appeal to users. By better understanding the epistemic heritage of QA, researchers, academia, and industry can more effectively accelerate QA research.

Research Talk

The video for our paper is available below:

Citation

If you cite our paper, please use this citation:

@inproceedings{rodriguez2021paradigms,
    title = "Evaluation Paradigms in Question Answering",
    author = "Pedro Rodriguez and Jordan Boyd-Graber",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    year = "2021",
    publisher = "Association for Computational Linguistics"
}

Contact

Please contact Pedro Rodriguez at me@pedro.ai