Spacy
This is an increasingly popular NLP library in Python that is powered by modern neural net-based models and features:
- tokenization;
- sentence segmentation;
- part-of-speech tagging;
- dependency parsing;
- named entity recognition;
- text classification;
- support of multiple languages;
- word/phrase similarity (between Glove vectors trained on Commoncrawl);
- GPU support;
and more…
https://spacy.io/

Today Facebook has opensourced PyText, which had been announced on the recent NeurIPS conference. PyText is a deep-learning based NLP modeling framework built on PyTorch. The library follows an extendable component-based design. PyText is used at Facebook for quick prototyping of new ideas and shipping them at scale. Some models:

- character-level text classifiers (inc. Y. Kim’s model);
- sequence taggers;
- intent-slot models for dialogue tasks.

https://github.com/facebookresearch/PyText

Happy holidays! I’m using this fascinating review of AI/ML advances by Xavier Amatriain (ex-Quora VP, ex-Netflix Director) as my concluding post in 2018. According to him, the key inventions of this year are:

- Interpretability of AI models: Causal Embeddings for Recommendations, High-Precision Model-Agnostic Explanations (Anchor/LIME);
- NLP-based systems: Google’s SmartCompose in GMail and Duplex dialog system;
- Language models and transfer learning for NLP tasks: UMLFit by Fast.ai, ELMO by Allen Institute, transformers by OpenAI, Google’s BERT;
- NLP frameworks: AllenNLP and FLAIR by Zalando;
- ML frameworks: PyTorch 1.0;
- Reinforcement learning: Impala by DeepMind, and frameworks: Google’s Dopamine and TRFL, Facebook’s Horizon, Microsoft’s TextWorld;
- Learning to rank (powered by deep learning): Google’s TFRank library;
- Data augmentation and training DL on synthetic data: Google’s auto-augment, Nvidia’s Training Deep Learning with Synthetic Data, and Snorkel from Stanford University.

https://medium.com/@xamat/the-year-in-ai-ml-advances-2018-roundup-db52f7f96358

☺️ https://twitter.com/boredyannlecun/status/1076993443052830721

Since Yann LeCun, Chief AI Scientist at Facebook and a co-recipient of the recent Turing Award, concluded the previous year in this channel, let him open this year. His article “Deep Learning Hardware: Past, Present, and Future” describes trends in deep learning research that will influence hardware and software of the future. While the whole paper is a fascinating read, I’ll give a few takeaways below:

- New architectural concepts, such as dynamic networks, differential programming, graph data/embeddings, memory networks, and complex inference are likely to affect the type of hardware architectures that will be required in the future;
- Future DL systems will largely be trained using a form of self-supervised learning. Such systems (e.g. video prediction with possible applications in self-driving cars) will eventually be trained to acquire vast amounts of background knowledge so as to acquire a form of common sense;
- The need of a new programming language that is, unlike Python, compilable, faster and more memory-efficient, interactive, dynamic, with safe parallelism, type inference, with support of important operations, such as convolutions;
- The author predicts very large networks with sparse activations, that new architectures could exploit for power reduction.

https://sci-hub.tw/https://doi.org/10.1109/ISSCC.2019.8662396

The recent release [1] of PyTorch-BigGraph (https://github.com/facebookresearch/PyTorch-BigGraph) from Facebook is a big thing for me. During the Ph.D. thesis, I worked closely with RESCAL and even came up with its more memory-efficient version (https://github.com/nzhiltsov/Ext-RESCAL) with an application in entity search. Since then, the graph embedding models (e.g. TransE, StarSpace) have improved a lot and the availability of their scalable implementations can boost many applications, such as link prediction (i.e. imputing missing relationships in knowledge bases) and nearest neighbors search in social networks.

Another somewhat relevant cutting edge research for building knowledge graphs automatically comes from M. Nickel, a co-creator of RESCAL. He, now at FAIR, and his colleagues investigate exploiting hyperbolic geometry of word embeddings to extract concept hierarchies from large text corpora [2]. Besides mathematical elegance, these embeddings come with a nice property of the learned representation: almost guaranteed transitivity.

Combining both sources, graph structure and text, always looked very promising to me. I predict to see many interesting advances and great applications in this area in the near future.

[1] https://ai.facebook.com/blog/open-sourcing-pytorch-biggraph-for-faster-embeddings-of-extremely-large-graphs/

[2] https://arxiv.org/abs/1902.00913

I’m at NAACL 2019 these days. “Transfer learning in NLP” is perhaps the most crowded tutorial.

Slides: http://tiny.cc/NAACLTransfer
Practice: https://colab.research.google.com/drive/1iDHCYIrWswIKp-n-pOg69xLoZO09MEgf

Many fascinating posters, e.g. a work by Alexey Romanov (UMass at Lowell) et al on transferring writing styles. Looking forward to the remaining days and especially the best paper section with BERT paper as a drawing card.

Dear colleagues, we at Orb Intelligence are publishing a series of posts about the most effective methods, techniques and frameworks used by academic researchers and industry engineers for different NLP tasks. Part I is about SOTA language representation learning methods. Your feedback is very welcome. #NLP #DeepLearning

https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-i-5f981c92af80

ACL 2019 has kicked off. The best time for the Part II of our NAACL 2019 highlights to be out: https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-ii-2a2a3dd42d1

DeepMind has launched its podcast 🚀 Find out more about cutting-edge AI, neuroscience, robotics and beyond firsthand:
🎙https://deepmind.com/blog/article/welcome-to-the-deepmind-podcast
🍏https://podcasts.apple.com/gb/podcast/deepmind-the-podcast/id1476316441

The NLP News newsletter by Sebastian Ruder (DeepMind, fast.ai) is yet another excellent source of relevant links (papers, blog posts, talks, slides, tools). I’m flattered that its last issue features our NAACL survey:
📧http://newsletter.ruder.io

GitHub has just launched a new NLP/IR challenge: CodeSearchNet challenge. The goal of code search is to retrieve relevant code given natural language.
They have released a dataset with:
- 99 queries, 4K relevance labels;
- 6m functions across 6 programming languages (Go, Java, Python etc)
- 2m out of 6m functions have the associated documentation (i.e. JavaDoc etc)
- And some metadata (line number and more).

They also published some baseline models (e.g. BERT-like self-attention).

📃Article: https://github.blog/2019-09-26-introducing-the-codesearchnet-challenge/
💻 Code: https://github.com/github/CodeSearchNet

Long-awaited Tensorflow 2.0 is out! Highlights:
- faster training/inference performance;
- tight integration with Keras;
- eager execution by default;
- standardized model persistence;
- distributed training and multi-GPU support;
- automatic migration script for TF 1.x

Official blog post: https://medium.com/tensorflow/tensorflow-2-0-is-now-available-57d706c2a9ab

I’m on “Neural Learning to Rank using TensorFlow Ranking: A Hands-on Tutorial” at ICTIR’19. Here’s the links to try TF-Ranking out:
GitHub: https://github.com/tensorflow/ranking
Colab: https://git.io/tf-ranking-demo

Snorkel 🐠

Labeling training data is a well-known bottleneck in developing machine learning-based applications. Snorkel is a system for programmatically (means quickly!) building and managing training datasets without manual labeling. It supports:
- labeling data, e.g., using heuristic rules or distant supervision techniques;
- transforming data, e.g., rotating or stretching images to perform data augmentation;
- slicing data into different critical subsets for monitoring or targeted improvement.

Snorkel models, cleans, and integrates the resulting training data using theoretically-grounded techniques (generative models, MCMC, Gibbs sampling).

Project website: https://www.snorkel.org

Snorkel at Google
📃Blog: https://ai.googleblog.com/2019/03/harnessing-organizational-knowledge-for.html
📃Article: https://arxiv.org/pdf/1812.00417.pdf