Security issue with system key:
-------------------------------
The configuration parameter that should be changed is sys_session_key
in /etc/gforge/local.inc.
 

local.inc:
------------
Add to /etc/gforge/local.inc
	
//URL to Nutch opensearch servlet
$sys_nutch_installation = 'http://nutch.nitrc.org:9080/nutch-0.9/opensearch';


Nutch Installation:
-------------------------

Nutch 0.9 is installed on int.nitrc.org.  Nutch 0.9 requires:

1. JDK 1.5 or higher
2. Tomcat 6 or higher

//download apache-ant:

root #: mkdir ant
root #: cd ant/

root #: wget http://apache.siamwebhosting.com/ant/binaries/apache-ant-1.7.1-bin.tar.gz
root #: tar -zxf apache-ant-1.7.1-bin.tar.gz 

root #: cd apache-ant-1.7.1
root #: cd bin/

//test ANT
root #: ant -version
root #: ./ant -version
root #: ./ant -diagnostics

root #: mkdir nutch
root #: cd nutch/
root #: svn checkout http://svn.apache.org/repos/asf/lucene/nutch/tags/release-0.9/


WEB CONFIGURATION FILES

Critical Configuration Files for are nutch-default.xml and nutch-site.xml. On int.nitrc.org, these files are located in /usr/local/tomcat/webapps/nutch-0.9/WEB-INF/classes/

- nutch-default.xml is the default configuration settings for nutch.  Also, it specifies the craw we are using.  
- nutch-site.xml is used to override the default settings in nutch-default.xml. Ideally this file should be modified as opposed to nutch-default.xml.  The location of the search index directory generated by the crawler should be specified in this file.  Likewise plugin configuration information would go here.

CRAWLER CONFIGURATION FILES

crawler-urlfilter.txt - Rules for filter urls would go here.  Also, needed to tell nutch to follow query string var???


TOMCAT

Tomcat should be restarted after configuration changes:
To restart Tomcat, go to /usr/local/tomcat/bin/ do the commands:
slide ./catalina start
slide ./catalina stop

export JRE_HOME:       /usr/java/jdk1.6.0_10
export CATALINA_BASE:   /usr/local/tomcat
export CATALINA_HOME:   /usr/local/tomcat
export CATALINA_TMPDIR: /usr/local/tomcat/temp
export JRE_HOME:       /usr/java/jdk1.6.0_10


After Tomcat and Nutch are installed:
----------------------------------------------------

 -  Get the custom plugin files from CVS:  
      /NITRCSRC/nutch/index-project, query-module, and query-project
 -  mv the files to the the nutch installation's src/plugins/ folder
 -  Get the updated nutch 0.9 files from CVS /NITRCSRC/nutch/src
 -  mv those files to the nutch/src folder
 -  in nutch/conf update crawl-urlfilter.txt to remove http limitation
    and comment #-.
 -  in nutch/conf update nutch-side.xml, nutch-default.xml
    - nutch-default also needs the custom plugins added
 -  recompile nutch: ant
 -  build new .war:  ant war
 -  cp the .war to tomcat's nutch web dir:  tomcat/webapps/
 -  restart Tomcat:  ./tomcat/bin/catalina.sh stop/start

 -  run a crawl:  bin/nutch crawl seed/urls -dir [new crawl dir name] -depth [n]
 -  restart Tomcat (again)



Install bot download activity remove cronjob
---------------------------------------------
(Note: Already done on production)
To the root's crontab add:
#
#       CLEAN BOT Activity from Download Statistics
#
0 4 * * * $PHP $GFORGE/cronjobs/remove_bot_downloads.php

