In this final week, we ended up choosing a text summarizer for our final project. We used a summarizer that uses sequence to sequence algorithm that uses bidirectional LSTM for its encoder, RNN and LSTM for the decoder using Bahdanau Attention. The project use TensorFlow, nltk, numpy, pandas, and langdetect for the library. The datasets fits just right, we trained and tested the model. We were pretty satisfied with the model.
We started to work on our final project. One library is not enough, so we experimented with a couple libraries that could be used for Natural Language Processing. TensorFlow has their own library for NLP, and there is a thing called TensorHub by TensorFlow. It is an open source library from google, and is mostly used as the embedding layer for NLP.
During this week, i was absent. Though, i learned about computer vision and the library used for it. We tried openCV for computer vision. My team worked with Ivan’s team for the computer vision task.
For our final project, we found a suitable datasets. It was on Kaggle with the name of all the news. It contains 143,000 articles from 15 news publisher. The datasets also has the label that we want, therefore we decided to use the dataset.
Link can be found below for the dataset.
In this week, we learned about Naive-Bayes classifier. We found out that it is one of the algorithm used for text classifier. We also followed the tutorial on medium on text classification with spacy, but the datasets used was too big. They tried using yelp reviews dataset and use only the spacy library to classify the text. We found out that it wasn’t suitable for our project. We started looking for datasets that we want to use for our final project
It was on week 6, we learned about Apriori. My team tried to work on the final project by testing out the code given on the previous article on medium. The first part show how spacy work on python and what it does. We tried summarizing a paragraph with spacy in accordance to the tutorial. It worked fine, but took quite some time to summarize the paragraph.
This week we learned about K-Means algorithm. Our team decided to create a text categorizer for the final project. We did some research and found out that doing that requires an understanding in Natural Language Processing. There are some blogs on medium written by one of the team working on Spacy, a python library for NLP.
https://medium.com/@ageitgey/natural-language-processing-is-fun-9a0bff37854e
The article is divided into 4 parts. It explained clearly what NLP is and how it works.