Word Vector Evaluation

Word Vectors


No. Vector Name Reference Short Description
1 Skip-Gram Mikolov et. al, 2013 Trained to be able to predict contextual words of a given word.
2 Cross-lingual Faruqui and Dyer, 2014 Cross-lingually enriched using canonical correlation across languages.
3 Glove Pennington et. al, 2014 Optimized to display linguistic regularities in language.
4 Hierarchical Yogatama et. al, 2015 Optimized to show hierarchical relations among dimensions of a vector.
5 Multi-sense Neelkantan et. al, 2014 Induce multiple sense specific vector representations for a single word type.
6 Global Context Huang et. al, 2012 Enriched using document level contextual information.
7 Non-distributional Faruqui and Dyer, 2015 Vectors constructed purely using linguistic knowledge with no distributional information.
8 MultiCluster and multiCCA Ammar et al, 2015 Massively multilingual word embeddings. Details can be found here.

Word Pair Similarity


No. Task Name Word pairs Reference
1 WS-353 353 Finkelstein et. al, 2002
2 WS-353-SIM 203 Agirre et. al, 2009
3 WS-353-REL 252 Agirre et. al, 2009
4 MC-30 30 Miller and Charles, 1991
5 RG-65 65 R and G, 1965
6 Rare-Word 2034 Luong et. al, 2013
7 MEN 3000 Bruni et. al, 2012
8 MTurk-287 287 Radinsky et. al, 2011
9 MTurk-771 771 Halawi and Dror, 2012
10 YP-130 130 Yang and Powers, 2006
11 SimLex-999 999 Hill et. al, 2014
12 Verb-144 144 Baker et. al, 2014

Word Relations


No. Task Name Size Description Reference
1 TOEFL 80 Choose closest synonym Landauer and Dumais, 1997
2 Syn-Rel 10765 Predict syntactic relation Mikolov et. al, 2013
3 Sem-Rel 8000 Predict semantic relation Mikolov et. al, 2013
4 SAT 374 Predict closest relation Turney et. al, 2003
5 Colors 52 Predict colors of nouns Bruni et. al, 2012
6 BLESS 26554 Predict relations b/w two words Baroni and Lenci, 2011
7 Phrase-Sim 5800 Predict similarity b/w two bigrams Mitchell and Lapata, 2008
8 Entailment 2770 Predict entailment b/w two words Baroni et al, 2012
9 SemEval-10 Task-8 8000 Predict similarity b/w two words in a sentence Hendrickx et al, 2010
10 SemEval-12 Task-2 79 Predict closest relation Jurgens et al, 2012
11 (Non-)Literal 342 Predict (non-)literal use in JJ-NN pairs Boleda et al, 2012

Other Languages


No. Language Task Name Size Reference
1 Arabic WS-353 353 Hassan and Mihalcea, 2009
2 Arabic MC-30 30 Hassan and Mihalcea, 2009
3 French WS-353 353 Joubarne and Inkpen, 2011
4 German RG-65 65 Gurevych, 2005
5 German Gur350 350 Gurevych, 2005
6 German ZG222 222 Zesch and Gurevych, 2006
7 Romanian WS-353 353 Hassan and Mihalcea, 2009
8 Romanian MC-30 30 Hassan and Mihalcea, 2009
9 Spanish WS-353 353 Hassan and Mihalcea, 2009
10 Spanish MC-30 30 Hassan and Mihalcea, 2009
11 Spanish RG-65 65 Camacho-Collados et al, 2015
12 Farsi RG-65 65 Camacho-Collados et al, 2015
13 de, it, ru WS-353, SimLex-999 varies Leviant and Reichart, 2015

Reference


If you use this website, please cite the following paper:
@InProceedings{faruqui-2014:SystemDemo,
  author    = {Faruqui, Manaal  and  Dyer, Chris},
  title     = {Community Evaluation and Exchange of Word Vectors at wordvectors.org},
  booktitle = {Proceedings of the 52nd Annual Meeting of the 
               Association for Computational Linguistics: System Demonstrations},
  month     = {June},
  year      = {2014},
  address   = {Baltimore, USA},
  publisher = {Association for Computational Linguistics}
}