full author list: Aarohi Srivastava and Abhinav Rastogi and Abhishek Rao and Abu Awal Md Shoeb and Abubakar Abid and Adam Fisch and Adam R. Brown and Adam Santoro and Aditya Gupta and Adri{\`a} Garriga-Alonso and Agnieszka Kluska and Aitor Lewkowycz and Akshat Agarwal and Alethea Power and Alex Ray and Alex Warstadt and Alexander W. Kocurek and Ali Safaya and Ali Tazarv and Alice Xiang and Alicia Parrish and Allen Nie and Aman Hussain and Amanda Askell and Amanda Dsouza and Ambrose Slone and Ameet Rahane and Anantharaman S. Iyer and Anders Johan Andreassen and Andrea Madotto and Andrea Santilli and Andreas Stuhlm{\"u}ller and Andrew M. Dai and Andrew La and Andrew Lampinen and Andy Zou and Angela Jiang and Angelica Chen and Anh Vuong and Animesh Gupta and Anna Gottardi and Antonio Norelli and Anu Venkatesh and Arash Gholamidavoodi and Arfa Tabassum and Arul Menezes and Arun Kirubarajan and Asher Mullokandov and Ashish Sabharwal and Austin Herrick and Avia Efrat and Aykut Erdem and Ayla Karaka{\c{s}} and B. Ryan Roberts and Bao Sheng Loe and Barret Zoph and Bart{\l}omiej Bojanowski and Batuhan {\"O}zyurt and Behnam Hedayatnia and Behnam Neyshabur and Benjamin Inden and Benno Stein and Berk Ekmekci and Bill Yuchen Lin and Blake Howald and Bryan Orinion and Cameron Diao and Cameron Dour and Catherine Stinson and Cedrick Argueta and Cesar Ferri and Chandan Singh and Charles Rathkopf and Chenlin Meng and Chitta Baral and Chiyu Wu and Chris Callison-Burch and Christopher Waites and Christian Voigt and Christopher D Manning and Christopher Potts and Cindy Ramirez and Clara E. Rivera and Clemencia Siro and Colin Raffel and Courtney Ashcraft and Cristina Garbacea and Damien Sileo and Dan Garrette and Dan Hendrycks and Dan Kilman and Dan Roth and C. Daniel Freeman and Daniel Khashabi and Daniel Levy and Daniel Mosegu{\'\i} Gonz{\'a}lez and Danielle Perszyk and Danny Hernandez and Danqi Chen and Daphne Ippolito and Dar Gilboa and David Dohan and David Drakard and David Jurgens and Debajyoti Datta and Deep Ganguli and Denis Emelin and Denis Kleyko and Deniz Yuret and Derek Chen and Derek Tam and Dieuwke Hupkes and Diganta Misra and Dilyar Buzan and Dimitri Coelho Mollo and Diyi Yang and Dong-Ho Lee and Dylan Schrader and Ekaterina Shutova and Ekin Dogus Cubuk and Elad Segal and Eleanor Hagerman and Elizabeth Barnes and Elizabeth Donoway and Ellie Pavlick and Emanuele Rodol{\`a} and Emma Lam and Eric Chu and Eric Tang and Erkut Erdem and Ernie Chang and Ethan A Chi and Ethan Dyer and Ethan Jerzak and Ethan Kim and Eunice Engefu Manyasi and Evgenii Zheltonozhskii and Fanyue Xia and Fatemeh Siar and Fernando Mart{\'\i}nez-Plumed and Francesca Happ{\'e} and Francois Chollet and Frieda Rong and Gaurav Mishra and Genta Indra Winata and Gerard de Melo and Germ{\'a}n Kruszewski and Giambattista Parascandolo and Giorgio Mariani and Gloria Xinyue Wang and Gonzalo Jaimovitch-Lopez and Gregor Betz and Guy Gur-Ari and Hana Galijasevic and Hannah Kim and Hannah Rashkin and Hannaneh Hajishirzi and Harsh Mehta and Hayden Bogar and Henry Francis Anthony Shevlin and Hinrich Schuetze and Hiromu Yakura and Hongming Zhang and Hugh Mee Wong and Ian Ng and Isaac Noble and Jaap Jumelet and Jack Geissinger and Jackson Kernion and Jacob Hilton and Jaehoon Lee and Jaime Fern{\'a}ndez Fisac and James B Simon and James Koppel and James Zheng and James Zou and Jan Kocon and Jana Thompson and Janelle Wingfield and Jared Kaplan and Jarema Radom and Jascha Sohl-Dickstein and Jason Phang and Jason Wei and Jason Yosinski and Jekaterina Novikova and Jelle Bosscher and Jennifer Marsh and Jeremy Kim and Jeroen Taal and Jesse Engel and Jesujoba Alabi and Jiacheng Xu and Jiaming Song and Jillian Tang and Joan Waweru and John Burden and John Miller and John U. Balis and Jonathan Batchelder and Jonathan Berant and J{\"o}rg Frohberg and Jos Rozen and Jose Hernandez-Orallo and Joseph Boudeman and Joseph Guerr and Joseph Jones and Joshua B. Tenenbaum and Joshua S. Rule and Joyce Chua and Kamil Kanclerz and Karen Livescu and Karl Krauth and Karthik Gopalakrishnan and Katerina Ignatyeva and Katja Markert and Kaustubh Dhole and Kevin Gimpel and Kevin Omondi and Kory Wallace Mathewson and Kristen Chiafullo and Ksenia Shkaruta and Kumar Shridhar and Kyle McDonell and Kyle Richardson and Laria Reynolds and Leo Gao and Li Zhang and Liam Dugan and Lianhui Qin and Lidia Contreras-Ochando and Louis-Philippe Morency and Luca Moschella and Lucas Lam and Lucy Noble and Ludwig Schmidt and Luheng He and Luis Oliveros-Col{\'o}n and Luke Metz and L{\"u}tfi Kerem Senel and Maarten Bosma and Maarten Sap and Maartje Ter Hoeve and Maheen Farooqi and Manaal Faruqui and Mantas Mazeika and Marco Baturan and Marco Marelli and Marco Maru and Maria Jose Ramirez-Quintana and Marie Tolkiehn and Mario Giulianelli and Martha Lewis and Martin Potthast and Matthew L Leavitt and Matthias Hagen and M{\'a}ty{\'a}s Schubert and Medina Orduna Baitemirova and Melody Arnaud and Melvin McElrath and Michael Andrew Yee and Michael Cohen and Michael Gu and Michael Ivanitskiy and Michael Starritt and Michael Strube and Micha{\l} Sw{\k{e}}drowski and Michele Bevilacqua and Michihiro Yasunaga and Mihir Kale and Mike Cain and Mimee Xu and Mirac Suzgun and Mitch Walker and Mo Tiwari and Mohit Bansal and Moin Aminnaseri and Mor Geva and Mozhdeh Gheini and Mukund Varma T and Nanyun Peng and Nathan Andrew Chi and Nayeon Lee and Neta Gur-Ari Krakover and Nicholas Cameron and Nicholas Roberts and Nick Doiron and Nicole Martinez and Nikita Nangia and Niklas Deckers and Niklas Muennighoff and Nitish Shirish Keskar and Niveditha S. Iyer and Noah Constant and Noah Fiedel and Nuan Wen and Oliver Zhang and Omar Agha and Omar Elbaghdadi and Omer Levy and Owain Evans and Pablo Antonio Moreno Casares and Parth Doshi and Pascale Fung and Paul Pu Liang and Paul Vicol and Pegah Alipoormolabashi and Peiyuan Liao and Percy Liang and Peter W Chang and Peter Eckersley and Phu Mon Htut and Pinyu Hwang and Piotr Mi{\l}kowski and Piyush Patil and Pouya Pezeshkpour and Priti Oli and Qiaozhu Mei and Qing Lyu and Qinlang Chen and Rabin Banjade and Rachel Etta Rudolph and Raefer Gabriel and Rahel Habacker and Ramon Risco and Rapha{\"e}l Milli{\`e}re and Rhythm Garg and Richard Barnes and Rif A. Saurous and Riku Arakawa and Robbe Raymaekers and Robert Frank and Rohan Sikand and Roman Novak and Roman Sitelew and Ronan Le Bras and Rosanne Liu and Rowan Jacobs and Rui Zhang and Russ Salakhutdinov and Ryan Andrew Chi and Seungjae Ryan Lee and Ryan Stovall and Ryan Teehan and Rylan Yang and Sahib Singh and Saif M. Mohammad and Sajant Anand and Sam Dillavou and Sam Shleifer and Sam Wiseman and Samuel Gruetter and Samuel R. Bowman and Samuel Stern Schoenholz and Sanghyun Han and Sanjeev Kwatra and Sarah A. Rous and Sarik Ghazarian and Sayan Ghosh and Sean Casey and Sebastian Bischoff and Sebastian Gehrmann and Sebastian Schuster and Sepideh Sadeghi and Shadi Hamdan and Sharon Zhou and Shashank Srivastava and Sherry Shi and Shikhar Singh and Shima Asaadi and Shixiang Shane Gu and Shubh Pachchigar and Shubham Toshniwal and Shyam Upadhyay and Shyamolima Shammie Debnath and Siamak Shakeri and Simon Thormeyer and Simone Melzi and Siva Reddy and Sneha Priscilla Makini and Soo-Hwan Lee and Spencer Torene and Sriharsha Hatwar and Stanislas Dehaene and Stefan Divic and Stefano Ermon and Stella Biderman and Stephanie Lin and Stephen Prasad and Steven Piantadosi and Stuart Shieber and Summer Misherghi and Svetlana Kiritchenko and Swaroop Mishra and Tal Linzen and Tal Schuster and Tao Li and Tao Yu and Tariq Ali and Tatsunori Hashimoto and Te-Lin Wu and Th{\'e}o Desbordes and Theodore Rothschild and Thomas Phan and Tianle Wang and Tiberius Nkinyili and Timo Schick and Timofei Kornev and Titus Tunduny and Tobias Gerstenberg and Trenton Chang and Trishala Neeraj and Tushar Khot and Tyler Shultz and Uri Shaham and Vedant Misra and Vera Demberg and Victoria Nyamai and Vikas Raunak and Vinay Venkatesh Ramasesh and vinay uday prabhu and Vishakh Padmakumar and Vivek Srikumar and William Fedus and William Saunders and William Zhang and Wout Vossen and Xiang Ren and Xiaoyu Tong and Xinran Zhao and Xinyi Wu and Xudong Shen and Yadollah Yaghoobzadeh and Yair Lakretz and Yangqiu Song and Yasaman Bahri and Yejin Choi and Yichi Yang and Yiding Hao and Yifu Chen and Yonatan Belinkov and Yu Hou and Yufang Hou and Yuntao Bai and Zachary Seid and Zhuoye Zhao and Zijian Wang and Zijie J. Wang and Zirui Wang and Ziyi Wu

This workshop is the fourth issue of a series of workshops on automatic extraction of socio-political events from news, organized by the Emerging Market Welfare Project, with the support of the Joint Research Centre of the European Commission and with contributions from many other prominent scholars in this field. The purpose of this series of workshops is to foster research and development of reliable, valid, robust, and practical solutions for automatically detecting descriptions of socio-political events, such as protests, riots, wars and armed conflicts, in text streams. This year workshop contributors make use of the state-of-the-art NLP technologies, such as Deep Learning, Word Embeddings and Transformers and cover a wide range of topics from text classification to news bias detection. Around 40 teams have registered and 15 teams contributed to three tasks that are i) multilingual protest news detection detection, ii) fine-grained classification of socio-political events, and iii) discovering Black Lives Matter protest events. The workshop also highlights two keynote and four invited talks about various aspects of creating event data sets and multi- and cross-lingual machine learning in few- and zero-shot settings.

In this paper, we describe our approach to utilize pre-trained BERT models with Convolutional Neural Networks for sub-task A of the Multilingual Offensive Language Identification shared task (OffensEval 2020), which is a part of the SemEval 2020. We show that combining CNN with BERT is better than using BERT on its own, and we emphasize the importance of utilizing pre-trained language models for downstream tasks. Our system, ranked 4th with macro averaged F1-Score of 0.897 in Arabic, 4th with score of 0.843 in Greek, and 3rd with score of 0.814 in Turkish. Additionally, we present ArabicBERT, a set of pre-trained transformer language models for Arabic that we share with the community.

End-to-end models trained on natural language inference (NLI) datasets show low generalization on out-of-distribution evaluation sets. The models tend to learn shallow heuristics due to dataset biases. The performance decreases dramatically on diagnostic sets measuring compositionality or robustness against simple heuristics. Existing solutions for this problem employ dataset augmentation which has the drawbacks of being applicable to only a limited set of adversaries and at worst hurting the model performance on other adversaries not included in the augmentation set. Instead, our proposed solution is to improve sentence understanding (hence out-of-distribution generalization) with joint learning of explicit semantics. We show that a BERT based model trained jointly on English semantic role labeling (SRL) and NLI achieves significantly higher performance on external evaluation sets measuring generalization performance.

We introduce Morse, a recurrent encoder-decoder model that produces morphological analyses of each word in a sentence. The encoder turns the relevant information about the word and its context into a fixed size vector representation and the decoder generates the sequence of characters for the lemma followed by a sequence of individual morphological features. We show that generating morphological features individually rather than as a combined tag allows the model to handle rare or unseen tags and outperform whole-tag models. In addition, generating morphological features as a sequence rather than e.g. an unordered set allows our model to produce an arbitrary number of features that represent multiple inflectional groups in morphologically complex languages. We obtain state-of-the art results in nine languages of different morphological complexity under low-resource, high-resource and transfer learning settings. We also introduce TrMor2018, a new high accuracy Turkish morphology dataset. Our Morse implementation and the TrMor2018 dataset are available online to support future research.

In this paper, we describe our system and results submitted for the Natural Language Inference (NLI) track of the MEDIQA 2019 Shared Task. As KU{\_}ai team, we used BERT as our baseline model and pre-processed the MedNLI dataset to mitigate the negative impact of de-identification artifacts. Moreover, we investigated different pre-training and transfer learning approaches to improve the performance. We show that pre-training the language model on rich biomedical corpora has a significant effect in teaching the model domain-specific language. In addition, training the model on large NLI datasets such as MultiNLI and SNLI helps in learning task-specific reasoning. Finally, we ensembled our highest-performing models, and achieved 84.7{\%} accuracy on the unseen test dataset and ranked 10th out of 17 teams in the official results.

Entity Extraction. Machine Learning. Named Entity Recognition. Natural Language Processing. Place Name Recognition. location="Seattle, WA, USA". articleno="8". numpages="10". acmid="3281363".

We introduce tree-stack LSTM to model state of a transition based parser with recurrent neural networks. Tree-stack LSTM does not use any parse tree based or hand-crafted features, yet performs better than models with these features. We also develop new set of embeddings from raw features to enhance the performance. There are 4 main components of this model: stack’s σ-LSTM, buffer’s β-LSTM, actions’ LSTM and tree-RNN. All LSTMs use continuous dense feature vectors (embeddings) as an input. Tree-RNN updates these embeddings based on transitions. We show that our model improves performance with low resource languages compared with its predecessors. We participate in CoNLL 2018 UD Shared Task as the ”KParse” team and ranked 16th in LAS, 15th in BLAS and BLEX metrics, of 27 participants parsing 82 test sets from 57 languages.

We present SParse, our Graph-Based Parsing model submitted for the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies (Zeman et al., 2018). Our model extends the state-of-the-art biaffine parser (Dozat and Manning, 2016) with a structural meta-learning module, SMeta, that combines local and global label predictions. Our parser has been trained and run on Universal Dependencies datasets (Nivre et al., 2016, 2018) and has 87.48% LAS, 78.63% MLAS, 78.69% BLEX and 81.76% CLAS (Nivre and Fang, 2017) score on the Italian-ISDT dataset and has 72.78% LAS, 59.10% MLAS, 61.38% BLEX and 61.72% CLAS score on the Japanese-GSD dataset in our official submission. All other corpora are evaluated after the submission deadline, for whom we present our unofficial test results.

25 September: http://cmmr2017.inesctec.pt/programme/scientific-programme/

We introduce context embeddings, dense vectors derived from a language model that represent the left/right context of a word instance, and demonstrate that context embeddings significantly improve the accuracy of our transition based parser. Our model consists of a bidirectional LSTM (BiLSTM) based language model that is pre-trained to predict words in plain text, and a multi-layer perceptron (MLP) decision model that uses features from the language model to predict the correct actions for an ArcHybrid transition based parser. We participated in the CoNLL 2017 UD Shared Task as the ``Ko\c{c} University'' team and our system was ranked 7th out of 33 systems that parsed 81 treebanks in 49 languages.

============================================================================ COLING 2016 Reviews for Submission #383 ============================================================================ Title: Learning grammatical categories using paradigmatic representations: Substitute words for language acquisition Authors: Mehmet Ali Yatbaz, Volkan Cirik, Aylin Küntay and Deniz Yuret ============================================================================ REVIEWER #1 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 2 Originality: 2 Technical correctness / soundness: 2 Readability and clarity: 2 Meaningful comparison: 2 Substance: 2 Impact of ideas: 2 Impact of resources: 1 Overall recommendation: 2 Reviewer Confidence: 3 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- This paper suggests that performance of the prediction of a syntactic category of a word in CHILDS corpora performs better with the model of the pattern with a slot (i.e. described as "a * b" in the article) than using a model separating the pattern heads ('aX') and the tails ('Xb') (i.e. described as "aX+Xb"). The performance of "a*b" being better than for "aX + Xb" in prediction seemed obvious to me in the presented experiment, since as in Figure 1, the information used for prediction in the former is larger than for the latter. In this sense, the comparison performed in this paper is not a "fair comparison". Maybe I misread, but if so, then authors must make sure to convince the readers that they are conducting a fair comparison. Therefore, I did not understand what claim is made in this article and my judgement with respect to COLING presentation unfortunately has to be reject. Apart from this main point, there are controversial issues related to this article. 1. The authors claim that "a*b" corresponds to "paradigmatic" representation, whereas "aX+Xb" to "syntagmatic", but uses of these two terms had better be reconsidered. In linguistics, paradigmatic and syntagmatic has the origin in F.de Saussure's definition. Calling "a*b"/"aX+Xb" as paradigmatic/syntagmatic seems too wide. 2. The authors refer to EMNLP-coNLL paper (the last reference), which has similar keywords in the title. If this refered paper is the authors' paper, then since the review process must be double blind, the authors should disclose the author names. If not, it must be shown what exactly is new of this paper compared with the reference. 3. The authors must explain more about their exact intent of this work. Do the authors want to engineer the CHILDES text? Then, the authors must explain the application needs and situate their work among previous work in the domain of engineering. Do the authors want to say from this paper, that children must be processing grammar through "a*b"? Then, the authors must explain the relation of their experiment how the experiment could relate to a cognitive claim. 4. Above all, I wonder whether this paper is appropriate for a presentation in the conferences of ACL community. Majority of the references are in the cognitive scientific domain. If behind this paper lies a cognitive question, then, it had better that they seek for a discussion in a conference dedicated in cognitive science domain. ============================================================================ REVIEWER #2 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 5 Originality: 3 Technical correctness / soundness: 4 Readability and clarity: 5 Meaningful comparison: 4 Substance: 3 Impact of ideas: 3 Impact of resources: 1 Overall recommendation: 4 Reviewer Confidence: 4 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- ## General comments This paper presents an interesting study about the use of paradigmatic presentations for word category learning in the setting of natural language acquisition. The authors compare two models of word representations -- syntagmatic (where each word is represented based on its neighbors), and paradigmatic (where the word is represented using substitute words). The experiments presented here clearly show the benefit of paradigmatic representations, and this is a very interesting contribution to the field. However, I'm having a hard time understanding the dichotomy presented between these two representations: both seem to rely on context-based substitutability criteria, since even the paradigmatic presentation constructs the substitute words via use of n-grams which in turn are created by looking at the neighboring words of each target word. A related point is that there is a slight misrepresentation of Harris's distributional hypothesis in the example presented in the Introduction: in Harris's view, substitutability arrives through the use of all (total) environments of the words/morphemes; it is therefore "unfair" to compare the substitutes which have been created from a very large collection of environments (n-grams) with the local context of these two sentences. ## Specific issues/comments The first sentence of your introduction would find an entire section of the field in disagreement, namely researchers who subscribe to the constructivist approach who believe that children learn rules about individual words and only later form abstract syntactic categories. After the example presented an introduction, you mentioned that "the high probability substitute reflect both semantic and grammatical properties". As many papers in the distributional semantic network area have shown, this is not necessarily the case especially if you select a very small window from which you create your context representations (something which applies to the n-grams you use in this work), in which case they mostly reflect syntactic properties. How do you justify the claim that the Redington et al. method "lacks completeness"? There are a few missing references from the part of speech induction literature that would be interesting to include: Alex Clark's 2003 "Combining distributional and morphological information for part of speech induction", Christodoulopoulos et. al 2011 "A Bayesian Mixture Model for PoS Induction Using Multiple Features" and Blunsom and Cohn's "A Hierarchical Pitman−Yor Process HMM for Unsupervised Part of Speech Induction". In the input corpora that you used, did you remove the punctuation marks? They tend to provide a very strong distributional cue and are sometimes explicitly removed to better simulate the linguistic input of children. It is unclear what the contribution of Figure 1 is. Is it theoretically better or worse to have more units as input to the model? Is there any evidence to support these theoretical claims? Given the size of the training corpus for the n-gram model, I would suggest that the comparison to the syntagmatic models which are trained on significantly fewer words seems problematic. I would've liked to see a per-tag graph of the number of substitutes for each target word, since I would think that for some cases of word classes (interjections) this number would return a lot of noise. It would be better if you describe the connection between the results that you get in this work and those of St. Clair et al. 2010. ============================================================================ REVIEWER #3 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 4 Originality: 4 Technical correctness / soundness: 4 Readability and clarity: 4 Meaningful comparison: 3 Substance: 3 Impact of ideas: 3 Impact of resources: 1 Overall recommendation: 4 Reviewer Confidence: 3 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- This paper describes a comparison of two approaches to POS tagging, one (syntagmatic) in which tags are assigned based on ngrams, and one (paradigmatic) in which tags are assigned based on possible lexical substitutions. The paper reports the latter (paradigmatic) gives higher accuracy. Overall it's an interesting result, but I have some reservations about how generally the results may be interpreted. First, if this model is to be taken as an acquisition model, as Section 1 seems to describe, I'm surprised the model uses supervised learning. Second, given this model is a supervised POS tagger, the accuracy results seem rather low compared to state-of-the-art sequence model taggers. This is potentially a problem because the gains due to paradigmatic modeling may not turn out to be significant in the presence of a more robust syntagmatic baseline. Minor comments: The last term in equation 3 should be "P(w_{n-1}|w^{n-2}_0)" (the zero is missing). Figures 2 and 3 should use bar graphs, since the slopes of lines between columns isn't interpretable. "patter(n)s" p. 5. You may want to cite work on syntagmatic and paradigmatic parser acquisition by Simon Dennis at U. Adelade in Australia.

COLING 2016 review: Title: CharNER: Character-Level Named Entity Recognition Authors: Onur Kuru, Ozan Arkan Can and Deniz Yuret ============================================================================ REVIEWER #1 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 5 Originality: 3 Technical correctness / soundness: 3 Readability and clarity: 4 Meaningful comparison: 4 Substance: 3 Impact of ideas: 3 Impact of resources: 3 Overall recommendation: 4 Reviewer Confidence: 4 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- The paper proposed character based NER for languages with word segmentation. The character-based tagging is proposed previously. Their contribution is to include LSTM models for the character-based tagging settings. However, the result is fair for the targeted languages. In my opinion, the method may be promising for the languages without word segmentation such as Chinese and Japanese, since the word segmentation error affects the score of NER. ============================================================================ REVIEWER #2 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 5 Originality: 3 Technical correctness / soundness: 4 Readability and clarity: 4 Meaningful comparison: 4 Substance: 3 Impact of ideas: 3 Impact of resources: 1 Overall recommendation: 4 Reviewer Confidence: 5 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- This paper presents a character-based model for named-entity recognition based on bidirectional LSTMs. There is very recent research that tries to do something similar, in NAACL-16 (Lample et al.) and ACL (Ma and Hovy, for example) with the exact same motivation: remove external resources, such as gazetteers and use a character-based approach to achieve high results. This should not invalidate the paper, though. The main difference with previous research (mentioned above) is that this model examines a sentence as a sequence of characters and outputs a tag distribution for each character. They later use transition matrices that only allow tags consistent with the word. The results are nice, however they are not the best at all. They are very good compared to systems that do not use external resources including word embeddings, however it should be a requirement to report results provided by the other systems without external resources (see Lample et al. for example) In Table 6, you present results for Ma and Hovy and Lample et al. and you include them in the "External" row, as far as I know they only use word embeddings (if they do)... I think that you should incorporate "word embeddings" to the caption, otherwise readers might think that they use gazetteers. Some missing references: Two EMNLP-15 papers that presented interesting results for tagging, parsing and language modeling by using character-based embeddings. - Wang Ling; Chris Dyer; Alan W Black; Isabel Trancoso; Ramon Fermandez; Silvio Amir; Luis Marujo; Tiago Luis Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation -Miguel Ballesteros; Chris Dyer; Noah A. Smith Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs This paper is also worth mentioning, since it also produces an entire character sequence for sentences: Bhuwan Dhingra; Zhong Zhou; Dylan Fitzpatrick; Michael Muehl; William Cohen Tweet2Vec: Character-Based Distributed Representations for Social Media Minor comment: Lample et al. do some more than LSTM-CRF, they also presented a shift reduce algorithm that exploits character-based embeddings. ============================================================================ REVIEWER #3 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 5 Originality: 3 Technical correctness / soundness: 4 Readability and clarity: 5 Meaningful comparison: 4 Substance: 4 Impact of ideas: 4 Impact of resources: 1 Overall recommendation: 4 Reviewer Confidence: 3 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- Very interesting work. It clearly shows that a deep bidirectional LSTM architecture combined with a Viterbi decoder effectively finds language specific features for NER. Applying the algorithm to Languages written without space characters, e.g., Chinese and/or Japanese, may be interesting. ================================= EMNLP 2016 review: Title: CharNER: Character-Level Named Entity Recognition Authors: Onur Kuru, Ozan Arkan Can and Deniz Yuret Instructions The author response period has begun. The reviews for your submission are displayed on this page. If you want to respond to the points raised in the reviews, you may do so in the box provided below. The response should be entered by 17 July, 2016 (11:59pm Pacific Daylight Savings Time, UTC -7h). Response can be edited multiple times during all the author response period. Please note: you are not obligated to respond to the reviews. Review #1 Appropriateness: 5 Clarity: 4 Originality: 3 Soundness / Correctness: 4 Meaningful Comparison: 5 Substance: 4 Impact of Ideas / Results: 4 Impact of Accompanying Software: 3 Impact of Accompanying Dataset / Resource: 1 Recommendation: 3 Reviewer Confidence: 5 Comments This paper presents a named entity recognizer in which the entire sentence is encoded as a sequence of characters, and a bidirectional LSTM is used to make predictions. Unlike previous (and recent) approaches, such as Lample et al. 2016 that presented character-based representation of words and then an LSTM/bidirectionalLSTM/stack-LSTM on top of that. This model is similar to the tweet2vec model recently accepted at ACL2016, even though it tries to solve a different task. They examine a sentence as a sequence of characters and outputs a tag distribution for each character.This model,as Lample et al., has the potentialities of being language independent and they apply it cross-lingually. The motivation and goals are also similar to Lample et al. (remove external features such as gazzetteers etc, and still achieve high results) Figure 3 does a great job summarizing the entire paper. In order to avoid things like J o h n w o r k s P O O O G G G G O they use a decoder as Wang et al. 2015, that applies a transition matrix, at the end they output the entire sequence. They make a good comparison with related work, but since some of the models are freely available, I'd expect that the authors run them in the languages without public results (such as Arabic or Turkish) In table 5 you should definitely differentiate between systems that use gazzetteers, and neural models that use pretrained word embeddings. They are not the same thing and how it is presented it might confuse the reader. This is an interesting paper, but it lacks a bit of novelty given all the previous work that already demonstrated the usefulness of characters and sequential models for NER. Minor comments: Missing ref (?) in related work.

Masked Reviewer ID: Assigned_Reviewer_2 Review: Question Overall rating Accept Reviewer expertise Some familiarity Paper strengths (2-3 brief points) Knet provides useful building blocks for efficiently implementing standard machine learning algorithms. The concise implementations for various algorithms provide performance competitive with other existing frameworks. The gradient descent method used in Knet allows for applications to provide code in an already existing high-level language. Paper weaknesses (2-3 brief points please) The paper does not clearly describe the limitations of the approach taken by Knet. Are there loss functions used in practice which cannot be handled by autograd? Are there algorithms used in practice which do not use gradient descent? The paper does not clearly state what functions are implemented or supplied by Knet. Is the grad function used in Section 2 just the same grad function supplied by autograd in Section 3.1? Detailed Comments How are the KnetArrays used by an application? Are the code segments in Section 2 using KnetArrays, or would the caller of "train" just supply arguments of type KnetArray? It would be useful to see larger scale benchmarks in Section 2.6. In addition, these benchmarks should describe whether a CPU or GPU is being used, which models of those, with how much memory, and which versions of the various pieces of software were tested. The code examples in Section 2 are clear and concise, which is very good, but some of the details of how the functions behave may be lost to readers unfamiliar with Julia (e.g., the array slicing syntax). In particular, the loss functions in 2.5 could be explained in a little more detail. The paper could be more clear if ordered slightly differently. Section 3 describes the functions provided by the knet library, which are used by the example implementations in Section 2. It may make more sense to swap this order. In 3.2, you mention that KnetArrays use a custom memory manager to reuse garbage collected pointers. This may be an interesting detail to elaborate on and evaluate the effects of (though in a short paper you may be space limited). Also in 3.2, you state that the paper examples run up to 17x faster on a GPU than a CPU. This number needs to be contextualized with the model of the CPU and the GPU you are using. Masked Reviewer ID: Assigned_Reviewer_3 Review: Question Overall rating Weak reject Reviewer expertise Some familiarity Paper strengths (2-3 brief points) The automatic differentiation approach seems like it could be an interesting mechanism, assuming the set of programs it works on are well characterized. It's nice to see higher level constructs applied to traditionally low-level environments like GPUs. Additionally, the code segments look like something a reasonable person could write. Paper weaknesses (2-3 brief points please) The contribution and motivation are unclear. It reads like the major contribution is the porting of an automatic differentiation module to Julia, along with some syntactic sugar over a CUDA library, though I assume there is more to it than that. Though the automatic differentiation technique looks promising, blackbox program transformations are slightly fraught in that they can fail on otherwise reasonable programs if they encounter a construct the author did not expect. Detailed Comments Overall, I'm slightly confused about who the paper's target audience is. Based on the examples in the paper, Knet targets those who wish to design their own models, i.e. people with a fairly sophisticated ML background. Yet for those much of the code in section 2 is redundant, a compelling example would simply take one non-trivial model and compare it against the next best framework. (Caffe is probably not the best best example to compare against, the decision to use a declarative rather than algorithmic approach is usually more fundamental than the choice of which library gets used). Furthermore, it's ambiguous what is actually being benchmarked; footnote 1 says the testbed uses AWS gpu instances but lines 262-263 imply that only CPU code was actually tested. Considering the importance of GPU targets this needs to be addressed preferably with a GPU comparison. Additionally, since TensorFlow is designed to function in a distributed setting comparing against it invites questions as to Knet's scalability. Since such a comparison is likely necessary, it may be a good idea to head off the issue by determining at what size dataset such scalability is actually beneficial. Also, since the benchmarks use different models, it would be good to have an indication that Knet based ones maintain rough parity for test data as well; as is there's no indication that they provide correct results. Additionally while automatic differentiation looks nifty, a cursory glance at the python Autograd's documentation implies the technique only works for a limited subset of the language's features. For a blackbox program transformation to be useable it really requires clear distinction between which language features work and which don't, ideally one that is automatically checked. Lacking this forces your users to retain a deep knowledge of which constructs are valid and which are not, and intuit when incorrect outputs are not due to their own bugs, but because of a mismatch between what they wrote and what the Knet backed can handle. It may be that you're already providing this, but if so that should be mentioned in the paper. Similarly, If you are achieving auto-vectorization, either through clever use of operator overloading, or from Julia's toolchain, this is a significant achievement and needs to be made explicit. As written, I cannot how much effort, if any, is required to translate between the example models and performant versions.

Title: Transfer Learning for Low-Resource Neural Machine Translation Authors: Barret Zoph, Deniz Yuret, Jonathan May and Kevin Knight Instructions The author response period has begun. The reviews for your submission are displayed on this page. If you want to respond to the points raised in the reviews, you may do so in the box provided below. The response should be entered by 17 July, 2016 (11:59pm Pacific Daylight Savings Time, UTC -7h). Response can be edited multiple times during all the author response period. Please note: you are not obligated to respond to the reviews. Review #1 Appropriateness: 5 Clarity: 4 Originality: 4 Soundness / Correctness: 4 Meaningful Comparison: 4 Substance: 4 Impact of Ideas / Results: 4 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 4 Reviewer Confidence: 4 Comments Neural machine translation (NMT) performs much worse than phrase-based or syntax-based statistical machine translation (SMT) for low-resource language pairs. This paper adopts transfer learning to improve the low-resource NMT. An NMT model is first trained on a rich-resource language pair, then transfer some learned parameters to the low-resource NMT model. In this way, the low-resource NMT obtained comparable performance as SMT. Although the idea is simple, experiments on four low-resource language pairs proved its efficiency. Moreover, a detailed and nice analysis is included in the paper, discussed the effect of the different rich-resource languages, fixing different part of the model, learning curve, etc. The section about key idea (section 3) lacks a bit of clarity, I understand the whole idea after read section 4 and 5, maybe you need to make it clearer what you exactly do transfer learning for NMT in section 3. A small question is related with this, do you use the same learning rate to train the parent model and child model? If not, how do you set the learning rate? Review #2 Appropriateness: 5 Clarity: 5 Originality: 3 Soundness / Correctness: 4 Meaningful Comparison: 4 Substance: 4 Impact of Ideas / Results: 3 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 3 Reviewer Confidence: 4 Comments The paper presents a transfer learning approach for low-resource NMT. The idea is extremely simple: training a full-scale parent model is used to initialise a child model with limited resources. The approach seems to be surprisingly effective and the authors convincingly present empirical proof for the use of this method. The final model is still below the performance of the baseline syntax-based SMT model in three out of four language pairs but outperforms the baseline when used for restoring n-bet lists. It seems surprising that the randomised mapping of word embeddings to the low-resource language word types actually works well enough. It would be interesting to know how much they are changed during training and how much they differ from training without parent model (if that can be quantified in some reasonable way). In the analyses part, the authors present a est in which they improve the mapping with dictionary-based assignment which sounds much more intuitive. But the use of this initialisation is minimal and seems to fade out completely in later training epochs. I'm still puzzled why there shouldn't be any effect of more appropriate mappings. Is there a problem with the dictionary-based assignments? Another question is whether this approach would also work the other way around. Could you fix the source language embeddings when translating from English into low-resource languages and would you expect the same kind of improvements? Did you already try this and would you have anything to report in the reverse direction? This would be more interesting then some of the ablation tests (like 5.2 which seems to argue for the same as 5.1) In any case, the discussion about the effect of parent language pair is limited as this does not systematically evaluate the effect using many language pairs and language families with various kinds of relationships. I have also a question about the syntax-based model used as baseline. What is the target syntax you are using in your string-to-tree model and how does it compare to state-of-the art settings? 26 BLEU for the English-Frech system sounds low to me but what is the test set and how does it compare to state-of-the-art? Another question would be about evaluations beyond BLEU scores. What does happen to the actual translations and what do humans think about them? The limits of automatic evaluation metrics are well know and the authors should discuss translation quality in other terms as well. Review #3 Appropriateness: 5 Clarity: 5 Originality: 3 Soundness / Correctness: 4 Meaningful Comparison: 5 Substance: 4 Impact of Ideas / Results: 4 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 4 Reviewer Confidence: 3 Comments The paper presents a method for transferring parameters from an NMT model learned on a large data set (fr-en) to a low resource translation task with the same output language (ha-en, tr-en, uz-en and ur-en). Part of the transfer process is selecting which parameters to fine-tune to the low resource translation task. Although I'm not convinced that the presented method is the best way to leverage the larger data set in a low resource translation setting, I strongly believe that this method needs to be presented to the community. In the prose you say that French' would be a good parent language for French, and I agree; in the experiment it seems like you flipped them so that French is the parent language, and French' is the child language. I don't know whether this makes any difference in your model at all, but please be consistent. In Table 7 you report ablation tests, measuring model perplexity and BLEU on the dev set. I'm left with the question of how these two quantities correlate with BLEU on an unseen test set. This question comes up again in the dictionary initialization experiments (Figure 4), where perplexity is again reported, but there is no way to tell how well it correlates with translation performance. Minor comments s3p2: "employing w a separate" -> "employing a separate" Fig1: the colors are really hard to see on a regular office laser printer print-out. Fig2: I'm guessing that eg "Source Word" and "Source Word" are different source words, and that they may be close to each other, but please use proper indexing to clarify exactly how.

Title: Why Neural Translations are the Right Length Authors: Xing Shi, Kevin Knight and Deniz Yuret Instructions The author response period has begun. The reviews for your submission are displayed on this page. If you want to respond to the points raised in the reviews, you may do so in the box provided below. The response should be entered by 17 July, 2016 (11:59pm Pacific Daylight Savings Time, UTC -7h). Response can be edited multiple times during all the author response period. Please note: you are not obligated to respond to the reviews. Review #1 Appropriateness: 5 Clarity: 5 Originality: 4 Soundness / Correctness: 5 Meaningful Comparison: 5 Substance: 4 Impact of Ideas / Results: 3 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 5 Reviewer Confidence: 4 Comments This paper convincingly explains why NMT translations are the right length; they show the mechanism (that there are components in the vectors that keep track of length during encoding and decoding) explicitly with a very clear toy example, as well as the tendency in a real-world task. The paper is admirably clearly written and very accessible. Please keep in mind that some people still print papers out, and stick to a gray scale legible coloring scheme (blue and red are indistinguishable after the gray scale dimensionality reduction has been applied). s1p1: "covert that vector in a target sentence." -> "convert that vector into a target sentence." Review #2 Appropriateness: 5 Clarity: 4 Originality: 4 Soundness / Correctness: 4 Meaningful Comparison: 4 Substance: 4 Impact of Ideas / Results: 3 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 4 Reviewer Confidence: 3 Comments This paper investigates the question of how neural MT models manage to produce output in the right length. Starting with a smal toy problem and proceeding to an actual neural MT system, the paper shows that (groups of) specific cells are "dedicated" to encoding sentence length. The paper is refreshingly different from many other NMT papers I've seen lately, in that it attempts to understand what's going on within the neural model, thus addressing a point of criticism that is occasionally brought forward against neural approaches to MT: that they are black boxes and no-one seems to care about what's going on inside. The paper is well written and generally easy to understand. What I'm missing most is a good motivation for addressing this question. What do we gain from knowing that neural models explicitly encode length? Also, is this behavior consistent across neural models? What happens if we start model training with different random initializations? A few minor quibbles: - The images in Figure 2 are two small. You've got plenty of room left (6 pages max.!) to provide bigger images. - Figures 4 and 5 should be tables, not figures. Review #3 Appropriateness: 5 Clarity: 5 Originality: 3 Soundness / Correctness: 5 Meaningful Comparison: 4 Substance: 4 Impact of Ideas / Results: 4 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 4 Reviewer Confidence: 4 Comments One of the issues with NMT is the apparent opacity of the model. It is hard to know what is going in inside the block box. The authors start to peel back the curtain here by investigating the question of how NMT models output target sentences of the right length. By looking at both a toy auto-encoder with 4 units and a regular NMT model, they provide interesting insight and show that the model has one of a small handful of units devoted specifically to the token counting task. The paper is well written & easy to follow and the insights will be of interest to the EMNLP audience.

Title: Context Embeddings using Substitute Word Distributions Authors: Volkan Cirik, Louis-Philippe Morency and Deniz Yuret Instructions The author response period has begun. The reviews for your submission are displayed on this page. If you want to respond to the points raised in the reviews, you may do so in the box provided below. The response should be entered by 17 July, 2016 (11:59pm Pacific Daylight Savings Time, UTC -7h). Response can be edited multiple times during all the author response period. Please note: you are not obligated to respond to the reviews. Review #1 Appropriateness: 5 Clarity: 5 Originality: 3 Soundness / Correctness: 4 Meaningful Comparison: 3 Substance: 3 Impact of Ideas / Results: 3 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 3 Reviewer Confidence: 5 Comments This work proposed to calculate context embeddings based on embeddings of substitute words. For a given context, a list of substitute words are firstly collected. Then, the weights of each substitute word is calculated based on a language model. Finally, the context embedding is computed by the weighed sum of all the substitute word embeddings. The author concatenate the context embedding with word embedding to represent a word under that context. Experiments on POS tagging task show the effectiveness of this method. The method is simple, and makes sense. However, I have some questions about the experiment part. (1) In Table 2, the number of features inputed to LIBSVM of the “word embedding” column is 50, whereas the number is 100 for the last column. Is it the reason that more features bring the higher performance? (2) Your word embeddings are pre-trained based on a much larger corpus. Is this the reason you got better performance in Table 4? (3) I’m wondering how will the result be if you only use the context embedding instead of concatenating it with the original word embedding. Review #2 Appropriateness: 5 Clarity: 4 Originality: 3 Soundness / Correctness: 5 Meaningful Comparison: 4 Substance: 4 Impact of Ideas / Results: 4 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 4 Reviewer Confidence: 4 Comments The paper presents an extension of (Yatbaz et al., 2012), which introduced substitute vectors for word context representations, and elegantly define so called context embeddings. The context embeddings possess a few interesting properties - they are adaptable to different word embeddings methods being thus universal in the input data too, they are naturally extending any word embedding method itself and, most importantly, they are able to improve the results of word embeddings by allowing to differentiate word sense occurrences. The improvement is documented in the paper by increased POS tagging accuracy of three word embeddings models and by state-of-the-art results for 5 unsupervised POS induction tasks on several languages (the other 5 results were comparable to the state-of-the-art). Comments and questions: The context of size 2n+1 of the substitute vectors uses only n-grams for the computation - why? Word2vec uses full word contexts for word embeddings, shouldn't they be used here too instead of Markov estimates? The text uses both p() and P() for probability - is there a difference? If not, they should be unified. Table 4 shows higher values of "Our method" in two more cases (Bulgarian CoNLL-X and Turkish CoNLL-X), these are, however, not bold. why? The related works could be expanded with other recent results of "context embeddings" computations with similar "independence" qualities, e.g. - Instance-context embeddings - Kågebäck, Mikael, et al. "Neural context embeddings for automatic discovery of word senses." Proceedings of NAACL-HLT. 2015. - Vu, Thuy, and D. Stott Parker. "K-Embeddings: Learning Conceptual Embeddings for Words using Context." Proceedings of NAACL-HLT. 2016. However, the context embeddings method from the current paper can be regarded as more transparent. Review #3 Appropriateness: 5 Clarity: 4 Originality: 3 Soundness / Correctness: 4 Meaningful Comparison: 2 Substance: 3 Impact of Ideas / Results: 3 Impact of Accompanying Software: 1 Impact of Accompanying Dataset / Resource: 1 Recommendation: 3 Reviewer Confidence: 4 Comments The paper describes an approach to learn embeddings of words in context and uses these embeddings to tackle POS tagging problems. It combines the concept of substitute word distributions (i.e. probability with which other words can replace a word in context) with traditional word embeddings (e.g. Collobert & Weston 2011). The contextual word embedding is the concatenation of the original embedding and the weighted sum of the top-K substitute word embeddings where the weights are determined by a statistical 4-gram language model. On supervised POS tagging the system achieves an accuracy of 96.7 which is close to state-of-the-art. On unsupervised POS tagging the system beats the state-of-the-art on 5 languages. Overall, the combination of n-gram language model with word embeddings to determine contextual embeddings seems novel. Although the supervised tagging results are not very positive, the unsupervised tagging results seem promising. However, it is unclear how impactful unsupervised tagging results are when typical downstream applications require a much higher accuracy that can be achieved through supervised or semi-supervised techniques. The other issue I had with the paper was that the way the contextual embeddings were inferred was unsatisfying. Instead of learning such embeddings from first principles, the embeddings were just the weighted sum of other words that appear in a similar context. In fact, the authors seem unaware of recent work in learning a different embedding for each sense of a given word (e.g. Iacobacci et. al. work in EMNLP'15 and others). It would be very helpful to compare against such approaches. Finally, the authors' claim that the state-of-the-art supervised POS tagging results requires hand-engineered features is misleading. There are several papers that use word embeddings and neural networks without hand-engineering to achieve >97.2% accuracy on the WSJ corpus (e.g. see Collobert & Weston 2011 or dos Santos et. al. ICML'14).

============================================================================ COLING 2016 Reviews for Submission #378 ============================================================================ Title: Recognizing Lexical Entailment using Substitutability Authors: Onur Kuru and Deniz Yuret ============================================================================ REVIEWER #1 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 5 Originality: 3 Technical correctness / soundness: 4 Readability and clarity: 4 Meaningful comparison: 5 Substance: 3 Impact of ideas: 3 Impact of resources: 1 Overall recommendation: 4 Reviewer Confidence: 5 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- This paper proposes a new solution for lexical entailment that brings together substitutability with entailment. While substitutability has always been tied in with the definition of lexical entailment, the authors claim that this is the first attempt to directly model that aspect. They compare their (unsupervised) approach to other existing approach, showing a respectable performance across different datasets and settings. Overall, I like this paper. It is clear to read, and uses simple ideas, but at the same time it tries to approach the problem from a slightly different point of view than existing work, and shows that this novel approach does fairly well. While lexical entailment cannot always be modeled by substitution (for example, when the word pair has different POS tags), there is definitely some advantage to tying the two together - and this paper does that much more explicitly than prior work. I have some (mostly minor, except point 1) concerns with this work though: 1) The most major concern is a factual error in Table 3 - the numbers for balAPinc evaluated on KDSZ in the different setting are incorrect (the Turney and Mohammad paper reports 0.60 AP1 and 0.60 AP0). This possibly invalidates some of the claims made in this paper regarding the efficacy of balAPinc v/s Subs. 2) Some of the comparisons in this paper are not apples-to-apples since the authors in this work do not deal with multi-word expressions and hence they have to work with only a subset of some of the datasets (specifically the comparisons on the KDSZ and the Zeichner datasets). 3) Since this is an unsupervised setup, I would have liked a bit more detail on the experimental setup. Is it k-fold cross validation, or do you just use a small subset of the dataset to tune thresholds? 3) The error analysis is not very insightful. Instead of reporting performance across corpora/# of tokens/n in n-gram, I would have liked to see an error analysis that is specific to the approach, along the lines of the "Substitute Distributions" sections. For example, what is it that makes this model better? Where does it do better than balAPinc? Are the things that balAPinc can get that the Subs model cannot get right? 4) There are some grammatical issues and typos. For instance : - Page 1, Second paragraph, the sentence starting "Since lexical entailment.." is not grammatical - Page 6, the line above table 3, "they only dependent" is incorrect ============================================================================ REVIEWER #2 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 5 Originality: 3 Technical correctness / soundness: 3 Readability and clarity: 4 Meaningful comparison: 4 Substance: 3 Impact of ideas: 2 Impact of resources: 1 Overall recommendation: 2 Reviewer Confidence: 4 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- This paper presents a rather simplistic scheme for inferring whether a lexical entailment between a pair of words hold. The approach is based on a probabilistic formulation of word substitutability is contexts where languages models are used to estimate probabilities of candidate words occurring in given contexts. For each context with a blank place holder, substitute distributions are computed for words occurring in that context by computing the probability of each word in that context and then normalizing. The proposed method has been tested on four data sets and Average Precision scores for both entail and does-not-entail scores have been computed. With the exception of one data set, the proposed approach performs better that other competing approaches based on WordNet. Additional Comments: -- What is the point of the equations 1-4 in section 2 given that they are not used /referred to later? (2) seems to have a typesetting problem: should n in the denominator be actually the summation bound? -- In your derivation of the P(b|a) approximation, I can probably buy the first assumption of a and b being independent. I understand why you need the second assumption mathematically but is that a reasonable assumption? It could very well be but you really need to provide a justification argument. Also in the final equation are C and C' varying of the same set of contexts? -- In references, please make sure your conference names are consistently named and capitalized (e.g. the first two), also Srilm -> SRILM; journal names should all be capital initial (e.g. Turney et al. and you should put all the authors without et al) ============================================================================ REVIEWER #3 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Relevance: 5 Originality: 3 Technical correctness / soundness: 3 Readability and clarity: 3 Meaningful comparison: 1 Substance: 4 Impact of ideas: 3 Impact of resources: 1 Overall recommendation: 3 Reviewer Confidence: 5 --------------------------------------------------------------------------- Comments --------------------------------------------------------------------------- This paper proposes to address the lexical entailment problem by directly modeling lexical substitutability: the word "dog" is assumed to entail the word "animal" if "animal" can replace "dog" in the contexts where "dog" occurs. This idea is closely related to the context inclusion hypothesis (dog entails animal if dog occurs in a subset of contexts where animal occurs), which has been used to design asymmetric distributional similarity functions] to detect lexical entailment. This paper proposes instead to use n-gram language models to directly score how often "dog" is a good replacement for "animal" in contexts drawn from corpora. This is an interesting idea which is presented clearly and with some positive empirical results. However, several modeling and experimental choices should be explained and motivated more thoroughly. In Section (3), what are the consequences of the simplifying assumption that context probabilities P(C) are uniform? This should be discussed since some contexts are clearly more likely than others. Experiments could be strengthened, and include controlled comparison with other asymmetric unsupervised methods (such as the approaches introduced in related work) beyond the random and (symmetric?) similarity baselines (Table 2). Results published elsewhere are reported for comparison (Table 3) in an out-of-domain evaluation setting, but it would be useful to see a comparison with controlled training conditions. Selecting good contexts to test how often "dog" can be substituted by "animal" seems to be a crucial step in the approach introduced here. This raises the question of how sensitive is the approach to the nature, domain, amount, diversity of contexts? Relatedly, what was the motivation for extracting contexts from the Reuters RCV1 dataset and using distinct corpora for language modeling? Other comments: In Equation (2), $n$ should be on top of the sum in the denominator. In Section (4), what does the FASTSUBS algorithm do?