Thursday, July 13, 2017

Setting up Apache Nutch v2.3.1 on Ubuntu with MongoDB and ElasticSearch

These are my notes for installing Apache Nutch v2.3.1 on Ubuntu in my work. We'll probably not use Nutch 2.3 because AWS ElasticSearch doesn't support the native transport protocol (it only supports the REST API, and it exposes it in a non-standard manner on port 80); thus we'll have a hard time indexing the data unless we maintain the servers ourselves. We can probably use AWS CloudSearch, but Nutch 2.3 doesn't have a plugin for it, unlike Nutch 1.13.

Anyway, there are lots of similar HOWTOs out there, but this one points to specific versions and URLs so you can just copy and paste these to get your system up and running in a few minutes.

OS used for this experiment: Ubuntu 16.04.2 LTS

Make sure that Ubuntu is the latest version

apt-get update
apt-get dist-upgrade
reboot #if necessary

Updating to the latest version is needed to fix the glibc bug that affects MongoDB. It’s probably not required anymore, but it’s good to be safe.

Install Java

add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java8-installer

Install ant

apt-get install ant

Increase the ulimit settings

(TODO: these settings should survive a reboot)
ulimit  -f unlimited \
        -t unlimited \
        -v unlimited \
        -n 64000 \
        -m unlimited \
        -u 64000

Install MongoDB

apt-key adv --keyserver hkp:// --recv 0C49F3730359A14518585931BC711F9BA15703C6

echo "deb [ arch=amd64,arm64 ] xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list

apt-get update
apt-get install mongodb-org

service mongod start

echo "use nutch" | mongo

Install Apache Nutch

Download and extract

tar xvf apache-nutch-2.3.1-src.tar.gz

Edit the configs

Uncomment this line at apache-nutch-2.3.1/ivy.xml:
<dependency org="org.apache.gora" name="gora-mongodb" rev="0.6.1" conf="*->default" />

Uncomment these lines at apache-nutch-2.3.1/runtime/local/conf/

Build Apache Nutch

ant runtime #needs to be run in the top directory

Install ElasticSearch

Setup ElasticSearch on AWS
Create a new AWS ElasticSearch domain and whitelist the server(s) where Apache Nutch is installed. AWS ElasticSearch doesn’t support the native transport protocol (port 9300). We may be able to use AWS if Nutch supports the HTTP REST protocol. Needs more research.

Manual Installation
# It's possible to use the latest version, ES5, but you need to update the indexer.

wget -qO - | sudo apt-key add -
echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list
apt-get update && sudo apt-get install elasticsearch

Edit /etc/elasticsearch/elasticsearch.yaml and put this: nutch nutch1

Finish up
update-rc.d elasticsearch defaults
/etc/init.d/elasticsearch restart

Install Kibana (optional, didn't install this since curl works fine)

echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list

apt-get update && sudo apt-get install kibana

Put this inside apache-nutch-2.3.1/runtime/local/conf/nutch-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <description>Default class for storing data</description>











No comments: