• May 29, 2024

Note: ElasticSearch, Kibana, FSCrawler on TrueNAS

When you finally realize that searching through decades of documents has become a challenge, it is time to acknowledge that you have reached a “big-data” stage. Now, you need some big data utilities to understand, index, search and find out where and what you’ve got.

This process is a nightmare. Bits of instructions are scattered across a half-dozen blogs, random posts and a mass of online instructions that are poorly described. This took me two days to get it working. These are my notes:

Overview

  1. Create a New Jail in TrueNas to hold and isolate the indexing system
  2. Configure the Jail
  3. Install ElasticSearch, Kibana, teseract
  4. Configure… Configure More… Ask why it is so complicated and disjoint
  5. Download the FSCrawler
  6. Configure more
  7. Make a script to run in background
  8. More configuring in Kibana
  9. Fix the massive memory demand issues

Create a new Jail in TrueNAS

  1. Login to your TrueNAS system and pick the Jails options
  2. Add a new Jail by clicking ADD
click “ADD” button
click “Advanced Jail Creation”
  1. Give it a Name: ElasticSearch
  2. Pick the newest Release you have installed
  3. Sect DHCP to get a unique IP address for this sub-system
  4. Select Auto-start
  5. Next
Enter Name in line 1
Select release to newest version
Click DHCP Autoconfig option box
Click Auto-Start opiton box
Click Next
  1. Select: allow_set_hostname
  2. Select: allow_raw_sockets
  3. Select: allow_mount
  4. Select: allow_mount_procfs
  5. Click: Next
  6. Click: Save
click option boxes:
allow_set_hosthome
allow_raw_sockets
allow_mount
click next
click save
  1. Wait for the jail creation to complete.
  2. Stop the Jail…

Configure the shared files you want to index by adding a Mount Point.

  1. Click on MOUNT POINTS
  2. Click on ACTIONS
  3. Click on ADD
  4. Select the host folder you want to index in the top panel
  5. Select /mnt for the location in the jail
  6. Select: Read-Only
  7. Click: Submit
in the jails control area, choose the ElasticSearch jail you just created, it expands a menu, click “mount points” option at the bottom of the panel
New option panel opens, click “actions” then “Add” in the drop down menu
new window opens, select source folder on the server, by expanding /mnt and picking your data folder.  Then, in the lower panel, attach to /mnt folder 
choose Read-only selection box
click submit
  1. Go to System Shell
click on the “Shell” option in the main menu on left side of screen

You need to change the StatFS option by using the command line. This allows the jail to see the running processes. We will use “iocage” to make setting changes to a jail. Use looks like this: iocage <get / set> <setting> <Name of Jail>

  1. Type: iocage get enforce_statfs ElasticSearch

It will show a value of 2

  1. This needs changed to a value of 1 by typing this: iocage set enforce_statfs=1 ElasticSearch
  1. Go Back to the Jails Section
  1. Start or Restart the Jail

Make Note of the IP address

Then Open the Shell for this Jail

  1. Start by installing the package manager and nano file editor. Type: pkg install nano
  2. It will prompt you to install pkg. Press “Y”
    It will then install pkg
    Then, it prompts you to install nano
  3. Press “Y”

Now we need to add the /proc mount point into the /etc/fstab

  1. type: nano /etc/fstab
  2. copy and paste this line into the new file:
    proc /proc procfs rw 0 0
  3. press control o – to save
  4. press enter to confirm file name
  5. press control x – to exit
  1. type mount /proc to attach the proc.

Install ElasticSearch, Kibana, Teseract

  1. To install elasticsearch, type: pkg install elasticsearch8
  2. press enter
  3. press “y” to acknowledge
  4. wait…
  1. update the rc.conf to allow elasticsearch to automatically start as a service
    Type: nano /etc/rc.conf
  2. move to the last line and copy/paste in this: elasticsearch_enable=“YES”
  3. press control+o
  4. enter to save
  5. press control+x to exit

Configure some of Elasticsearch
The config files are stored in /usr/local/etc/elastic search

  1. type: nano /usr/local/etc/elasticsearch/elasticsearch.yml

Now we make changes to the name of the elastic cluster:

  1. Uncomment the line the says cluster.name: and change the name to whatever you want
  2. Uncomment the node.name and change as needed
  1. Scroll down to the Network Section
  2. uncomment the network.host: line
  3. update the ip address with the address of your system – which you wrote down earlier to avoid having to exit the editor/shell and looking for it.
  4. press control+o
  5. press enter to save
  6. press control+x to exit
  1. start elasticsearch by typing: service elasticsearch start

It should eventually start….

  1. Now we reset the password with this command: elasticsearch-reset-password -u elastic
  2. press “y”
  3. It will print out the password on the screen. Write your new password down or copy it out for use later!
  1. Reset the password for the Kibana system user by typing: elasticsearch-reset-password -u kibana_system
  2. press “y”
  3. Write down the password for the kibana_system user
  1. Now we create basic TLS security by typing: elasticsearch-certutil ca
  2. Press enter.
  3. You can add passwords here if needed and/or just press enter to leave blank
    More information is available here
  4. Now, type: elasticsearch-certutil cert –ca elastic-stack-ca.p12
  5. Press enter a few times and/or add passwords as needed

If you entered a password… more steps are needed.
Enter this: elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
Enter this: elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

This section also automatically updated the configuration file at /usr/local/etc/elasticsearch/elasticsearch.yml

Install Kibana

  1. Type: pkg install kibana8
  2. press enter
  3. press “y”

Edit the rc.conf file to autostart Cuban

  1. Type: nano /etc/rc.conf
  2. press enter
  3. scroll to the bottom of the file and add or copy/pate this: kibana_enable=”YES”
  4. press control+o to save
  5. press enter
  6. press control+x to exit
  1. Now, edit the kibana config file by typing: nano /usr/local/etc/kibana/kibana.yml
  2. press enter
  3. uncomment the server.host: line
  4. type in your jail’s IP address
  1. Start Kibana by typing: service kibana start
  2. Use the browser to open a new tab or window and navigate to the kibana address.
    In this case it is http://192.168.1.234:5601
    Replace with your jail’s ip address
    The site should look like this:
  1. Now, go back the jail’s shell terminal and fix a write issue.
    The kibana.yml config file is write locked and the online enrollment will fail. The file permissions need changed for the online enrollment to work.
    Type: chmod 777 /usr/local/etc/kibana/kibana.yml
  2. press enter
  3. Now, we make the token for the web setup for kibana by typing:
    elasticsearch-create-enrollment-token –scope kibana
  4. Press enter
  5. It will give you a code to copy and paste back into kibana browser window:
  1. Press: Configure Elastic
    It will then ask you for a verification code
  1. Go back to the jail’s shell and copy/paste the command: kibana-verification-code
  2. Press enter
  3. Note the code to type in the box. Type in code and click: “Verify”

Wahoo!

You can now log in with the elastic account that you created/reset the password for way back when…

Let’s setup monitoring.

  1. Click on the menu on the left – the three lines
  2. Go to the bottom of the menu and click Stack Monitoring
  3. Click: “Or, set up with self monitoring”
  4. Click: “Turn on Monitoring”
  5. Wait for it to finish

Make sure the everything is good!

Install FSCrawler and Tesseract OCR

  1. type: pkg install wget
  2. press enter
  3. press “y”

The FSCrawler webiste is here

We need the location of the newest file: Located here. Note: Only the latest version has the ElasticSearch version 8 extensions. The other older versions will not work. You need the link of the latest one… at the bottom of the page… right click and copy link

  1. Go back in the jail’s shell type this: cd
  2. press enter
    ~you should be back at the root home folder~
  3. Type: wget https://s01.oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-distribution/2.10-SNAPSHOT/fscrawler-distribution-2.10-20231023.160816-291.zip
  4. press enter
  5. it will download the files
  6. type: unzip fscrawler-distribution-2.10-20231023.160816-291.zip
    This extracts the files
  7. If you type ls and press enter. You should see the new files:
  1. for ease of typing… rename the folder by typing this:
    mv fscrawler-distribution-2.10-SNAPSHOT fscrawler

Now… we initiate the first run of the fscrawler to create basic setting file for us to edit

  1. type: ./fscrawler/bin/fscrawler scanfiles
  2. press enter
  3. type “y” to create
  4. press enter
  5. open the newly created settings file to edit by typing: nano .fscrawler/scanfiles/_settings.yaml
  6. edit the url line to indicate the location of the files which are now appearing in /mnt
    url: “/mnt”
  7. edit the elasticsearch: nodes: url: to show your jail’s ip address: 192.168.1.234
  8. add a line and type: username: “elastic”
  9. add a line and type: password: “bZb0v7BlO38sWyB0a1M7”
    use your password here from when you reset it earlier and wrote it down
  10. we are not doing SSL verification
  11. press control+o to save
  12. press enter
  13. press control+x to exit
  1. Install tesseract OCR package by typing: pkg install tesseract
  2. press enter
  3. press “Y”
  4. wait for it to finish
  5. See if it all starts okay by typing the following: ./fscrawler/bin/fscrawler scanfiles

Set a Cron to Schedule the FSCrawler

  1. type crontab -e
  2. press enter
  3. The screen enters a vi editor, press “i” to enter edit mode
  4. set the runtime by using @600 for running every 10 minutes. Type: @600
  5. then type the command /root/fscrawler/bin/fscrawler scanfiles
  6. press esc
  7. press :w
  8. press enter
  9. press :
  10. press q
  11. press enter

The search will run on the new schedule and create an index for you to search in Kibana

Fix the memory Issue

ElasticSearch, Kibana, and FScrawler are all running in Java virtual machines that will each will take up to 50% of the total system memory. I’m sure this made some sense to someone… THIS WILL CRASH your TrueNAS server. You need enough memory for ZFS to work, and if you accidentally decide to do anything, poof… system halt… process protection starts to kill off tasks that you might actually need.

Now we tweak the memory footprint for the JVM.

We are going to edit the system wide environment variables. In FreeBSD these are stored in the csh settings file

  1. Type: nano /etc/csh.cshrc
  2. We can add a line to set the JAVA_HOME. Type: setenv JAVA_HOME /usr/local/openjdk17
  3. Add a line to fix the JAVA Memory. This has 2 parts, the starting (Xms) and max (Xmx)
    I’m picking approx 2 GB of memory use for each instance, that is 2 GB x 3 processes = 6 GB.
    Type: setenv FS_JAVA_OPTS “-Xmx2048m -Xms2048m”
  4. ctrl + o
  5. enter to save
  6. ctrl +x
  7. type: exit
  8. press enter
  9. You should be back at the jail’s control panel, press the “restart” button

DONE!

At this point, the jail system should reboot and start without any errors.
It will take a few moments for the servers to all start-up. Then, you can then go to Kibana dashboard and monitor the progress, do a search, look through your index.

The cron scheduler will start FSCrawler in a few minutes. It will first index the folders, and then start on the files. You may want to modify the settings in the fscrawler settings further to do OCR, not do OCR, exclude some files, etc.

Keep in mind… depending on how many files and the complexity, it might take days to complete.