Setting up Apache Nutch v2.3.1 on Ubuntu with MongoDB and ElasticSearch

These are my notes for installing Apache Nutch v2.3.1 on Ubuntu in my work. We'll probably not use Nutch 2.3 because AWS ElasticSearch doesn't support the native transport protocol (it only supports the REST API, and it exposes it in a non-standard manner on port 80); thus we'll have a hard time indexing the data unless we maintain the servers ourselves. We can probably use AWS CloudSearch, but Nutch 2.3 doesn't have a plugin for it, unlike Nutch 1.13.

Anyway, there are lots of similar HOWTOs out there, but this one points to specific versions and URLs so you can just copy and paste these to get your system up and running in a few minutes.

OS used for this experiment: Ubuntu 16.04.2 LTS

Make sure that Ubuntu is the latest version

apt-get update

apt-get dist-upgrade

reboot #if necessary

Updating to the latest version is needed to fix the glibc bug that affects MongoDB. It’s probably not required anymore, but it’s good to be safe.

Install Java

add-apt-repository ppa:webupd8team/java

apt-get update

apt-get install oracle-java8-installer

Install ant

apt-get install ant

Increase the ulimit settings

(TODO: these settings should survive a reboot)

ulimit  -f unlimited \

        -t unlimited \

        -v unlimited \

        -n 64000 \

        -m unlimited \

        -u 64000

Install MongoDB

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6




echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list




apt-get update

apt-get install mongodb-org




service mongod start




echo "use nutch" | mongo

Install Apache Nutch

Download and extract

wget http://apache.cs.utah.edu/nutch/2.3.1/apache-nutch-2.3.1-src.tar.gz

tar xvf apache-nutch-2.3.1-src.tar.gz

Edit the configs

Uncomment this line at apache-nutch-2.3.1/ivy.xml:

<dependency org="org.apache.gora" name="gora-mongodb" rev="0.6.1" conf="*->default" />

Uncomment these lines at apache-nutch-2.3.1/runtime/local/conf/gora.properties:

gora.datastore.default=org.apache.gora.mongodb.store.MongoStore

gora.mongodb.override_hadoop_configuration=false

gora.mongodb.mapping.file=/gora-mongodb-mapping.xml

gora.mongodb.servers=localhost:27017

gora.mongodb.db=nutch

Build Apache Nutch

ant runtime #needs to be run in the top directory

Install ElasticSearch

Setup ElasticSearch on AWS

Create a new AWS ElasticSearch domain and whitelist the server(s) where Apache Nutch is installed. AWS ElasticSearch doesn’t support the native transport protocol (port 9300). We may be able to use AWS if Nutch supports the HTTP REST protocol. Needs more research.

Manual Installation

# It's possible to use the latest version, ES5, but you need to update the indexer.




wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list

apt-get update && sudo apt-get install elasticsearch

Edit /etc/elasticsearch/elasticsearch.yaml and put this:

network.host: 127.0.0.1

cluster.name: nutch

node.name: nutch1

Finish up

update-rc.d elasticsearch defaults

/etc/init.d/elasticsearch restart

Install Kibana (optional, didn't install this since curl works fine)

echo "deb http://packages.elastic.co/kibana/4.4/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list




apt-get update && sudo apt-get install kibana

Put this inside apache-nutch-2.3.1/runtime/local/conf/nutch-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>




<configuration>

  <property>

    <name>storage.data.store.class</name>

    <value>org.apache.gora.mongodb.store.MongoStore</value>

    <description>Default class for storing data</description>

  </property>




  <property>

    <name>http.agent.name</name>

    <value>Spiderman</value>

  </property>




  <property>

    <name>plugin.includes</name>

    <value>protocol-(http|httpclient)|urlfilter-regex|index-(basic|more)|query-(basic|site|url|lang)|indexer-elastic|nutch-extensionpoints|parse-(text|html|msexcel|msword|mspowerpoint|pdf)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)|parse-(html|tika|metatags)|index-(basic|anchor|more|metadata)</value>

  </property>

  <property>

    <name>elastic.host</name>

    <value>127.0.0.1</value>

  </property>




  <property>

    <name>elastic.port</name>

    <value>9300</value>

  </property>




  <property>

    <name>elastic.cluster</name>

    <value>nutch</value>

  </property>




  <property>

    <name>elastic.index</name>

    <value>nutch</value>

  </property>




  <property>

    <name>parser.character.encoding.default</name>

    <value>utf-8</value>

  </property>




  <property>

    <name>http.content.limit</name>

    <value>6553600</value>

  </property>




  <property>

    <name>elastic.max.bulk.docs</name>

    <value>2000</value>

  </property>




  <property>

    <name>elastic.max.bulk.size</name>

    <value>2500500</value>

  </property>




</configuration>