The AI Digest

A topic model is a type of statistical model for discovering the abstract “topics” that occur in a collection of documents. Topic modeling is a frequently used text-mining tool for the discovery of hidden semantic structures in a text body. It is an unsupervised technique to know which “topic” a text document belongs to. See how to do topic modeling using Roberta and transformers. We will use a pre-trained Roberta model finetuned on the NLI dataset for getting embeddings and then do topic modelling.

DialoGPT from Microsoft is a SOTA large-scale pretrained dialogue response generation model for multiturn conversations. The responses generated by this model is comparable to human response quality under a single-turn conversation Turing test. See how to do conversational response generation using DialoGPT.

ONNX stands for Open Neural Network Exchange. It is a cross-platform inferencing and training accelerator compatible with many popular ML/DL frameworks like PyTorch, Tensorflow, Scikit-learn. See how you can use ONNX along with Huggingface transformers pipeline for faster inference of NLP tasks.

How to do Question answering using Huggingface transformers and BERT? See how you can use the transformers pipeline for Question answering using BERT.

Now hugging face transformers and sentence-transformers made it easy to cluster documents using BERT embeddings. basically, you need to get the embedding for your text using BERT and then apply K-means clustering on these embeddings. You can follow the steps mentioned in my blog.

How to do semantic document similarity using BERT. Two basic steps, first get the embedding of documents using BERT and then calculate the cosine similarity score between them. Similar documents will have high similarity score. Check my latest blog post for the code.

Learn how to do zero-shot classification of text using the Huggingface transformers pipeline. Also, see where it fails and how to resolve it.

https://theaidigest.in/zero-shot-classification-using-huggingface-transformers-pipeline/

Summarize text document using Huggingface transformers and BERT. Use different transformer models for summary and findout the performance.

Elasticsearch has a very convenient python package called “elasticsearch” to extract data from Elasticsearch. Getting data from Elasticsearch makes it easier for data analysis in python. 'Scan' is part of the elasticsearch helpers function. It is used to get data if your response has more than 10K documents. In this blog post I explained how to use scan function to get data from elasticsearch using Python. This works even if you have more than 10K documents in your index.

Kibana has a convenient tool called “dev tools” to extract data from Elasticsearch and interact with it. Dev tools has a tab called 'Console'. Here you can write elasticsearch queries and see the response in the right side pane. It also has 'Search profiler', which show shows the performance of the query you ran like total time taken. Please see my new blog post on how to use use Kibana dev tools for getting data from elasticsearch.

How to load a CSV file into Elasticsearch using python? Well, its very simple using the Elasticsearch python package.1. Install elasticsearch.2. Acquire csv dataset.3. Check if elasticsearch cluster is up and running.4. Install elasticsearch python package.5. Bulk index CSV file into elasticsearch using the elasticsearch python package.6. Do post indexing checkup like checking if index created and you can see data.Details steps and code in my website. 

https://theaidigest.in/load-csv-into-elasticsearch-using-python/

Do you know how does elasticsearch scoring works? Lucene/Elasticsearch uses the Boolean model to find matching documents, and a formula called the practical scoring function to calculate the score. Here I have explained in detail how the scoring works.

https://theaidigest.in/how-does-elasticsearch-scoring-work/

Elasticsearch is a JSON document store built upon the Apache Lucene search engine. It is a search engine for different data types like text, numeric, geospatial, structured, and unstructured data. Learn in detail here.

https://theaidigest.in/what-is-elasticsearch/

Kibana is an open source data visualization dashboard for data stored in Elasticsearch. It visualizes the content indexed on an Elasticsearch cluster using bar, line and scatter plots, or pie charts and maps on top of large volumes of data. Here is how you can install kibana.

https://theaidigest.in/install-kibana-in-ubuntu-20-04/

Graylog is a centralized log management tool built for capturing, storing, and enabling real-time analysis of terabytes of machine data. Here is how you can install graylog in Ubuntu 20.04

Elasticsearch is a distributed, open-source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is fast, distributed by nature. Here you can find steps to install elasticsearch.