Explore, Create & Enjoy

Note: ElasticSearch, Kibana, FSCrawler on TrueNAS

• December 17, 2023February 25, 2024

When you finally realize that searching through decades of documents has become a challenge, it is time to acknowledge that you have reached a “big-data” stage. Now, you need some big data utilities to understand, index, search and find out where and what you’ve got.

This process is a nightmare. Bits of instructions are scattered across a half-dozen blogs, random posts and a mass of online instructions that are poorly described. This took me two days to get it working. These are my notes:

Overview

1. Create a New Jail in TrueNas to hold and isolate the indexing system
2. Configure the Jail
3. Install ElasticSearch, Kibana, teseract
4. Configure… Configure More… Ask why it is so complicated and disjoint
5. Download the FSCrawler
6. Configure more
7. Make a script to run in background
8. More configuring in Kibana
9. Fix the massive memory demand issues

Create a new Jail in TrueNAS

1. Login to your TrueNAS system and pick the Jails options
2. Add a new Jail by clicking ADD

3. Give it a Name: ElasticSearch
4. Pick the newest Release you have installed
5. Sect DHCP to get a unique IP address for this sub-system
6. Select Auto-start
7. Next

8. Select: allow_set_hostname
9. Select: allow_raw_sockets
10. Select: allow_mount
11. Select: allow_mount_procfs
12. Click: Next
13. Click: Save

14. Wait for the jail creation to complete.
15. Stop the Jail…

Configure the shared files you want to index by adding a Mount Point.

16. Click on MOUNT POINTS
17. Click on ACTIONS
18. Click on ADD
19. Select the host folder you want to index in the top panel
20. Select /mnt for the location in the jail
21. Select: Read-Only
22. Click: Submit

23. Go to System Shell

You need to change the StatFS option by using the command line. This allows the jail to see the running processes. We will use “iocage” to make setting changes to a jail. Use looks like this: iocage <get / set> <setting> <Name of Jail>

24. Type: iocage get enforce_statfs ElasticSearch

It will show a value of 2

25. This needs changed to a value of 1 by typing this: iocage set enforce_statfs=1 ElasticSearch

27. Go Back to the Jails Section

28. Start or Restart the Jail

Make Note of the IP address

Then Open the Shell for this Jail

29. Start by installing the package manager and nano file editor. Type: pkg install nano
30. It will prompt you to install pkg. Press “Y”
    It will then install pkg
    Then, it prompts you to install nano
31. Press “Y”

Now we need to add the /proc mount point into the /etc/fstab

33. type: nano /etc/fstab
34. copy and paste this line into the new file:
    proc /proc procfs rw 0 0
35. press control o – to save
36. press enter to confirm file name
37. press control x – to exit

38. type mount /proc to attach the proc.

Install ElasticSearch, Kibana, Teseract

39. To install elasticsearch, type: pkg install elasticsearch8
40. press enter
41. press “y” to acknowledge
42. wait…

42. update the rc.conf to allow elasticsearch to automatically start as a service
    Type: nano /etc/rc.conf
43. move to the last line and copy/paste in this: elasticsearch_enable=“YES”
44. press control+o
45. enter to save
46. press control+x to exit

Configure some of Elasticsearch
The config files are stored in /usr/local/etc/elastic search

47. type: nano /usr/local/etc/elasticsearch/elasticsearch.yml

Now we make changes to the name of the elastic cluster:

48. Uncomment the line the says cluster.name: and change the name to whatever you want
49. Uncomment the node.name and change as needed

50. Scroll down to the Network Section
51. uncomment the network.host: line
52. update the ip address with the address of your system – which you wrote down earlier to avoid having to exit the editor/shell and looking for it.
53. press control+o
54. press enter to save
55. press control+x to exit

56. start elasticsearch by typing: service elasticsearch start

It should eventually start….

57. Now we reset the password with this command: elasticsearch-reset-password -u elastic
58. press “y”
59. It will print out the password on the screen. Write your new password down or copy it out for use later!

60. Reset the password for the Kibana system user by typing: elasticsearch-reset-password -u kibana_system
61. press “y”
62. Write down the password for the kibana_system user

63. Now we create basic TLS security by typing: elasticsearch-certutil ca
64. Press enter.
65. You can add passwords here if needed and/or just press enter to leave blank
    More information is available here
66. Now, type: elasticsearch-certutil cert –ca elastic-stack-ca.p12
67. Press enter a few times and/or add passwords as needed

If you entered a password… more steps are needed.
Enter this: elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
Enter this: elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

This section also automatically updated the configuration file at /usr/local/etc/elasticsearch/elasticsearch.yml

Install Kibana

68. Type: pkg install kibana8
69. press enter
70. press “y”

Edit the rc.conf file to autostart Cuban

71. Type: nano /etc/rc.conf
72. press enter
73. scroll to the bottom of the file and add or copy/pate this: kibana_enable=”YES”
74. press control+o to save
75. press enter
76. press control+x to exit

77. Now, edit the kibana config file by typing: nano /usr/local/etc/kibana/kibana.yml
78. press enter
79. uncomment the server.host: line
80. type in your jail’s IP address

81. Start Kibana by typing: service kibana start
82. Use the browser to open a new tab or window and navigate to the kibana address.
    In this case it is http://192.168.1.234:5601
    Replace with your jail’s ip address
    The site should look like this:

83. Now, go back the jail’s shell terminal and fix a write issue.
    The kibana.yml config file is write locked and the online enrollment will fail. The file permissions need changed for the online enrollment to work.
    Type: chmod 777 /usr/local/etc/kibana/kibana.yml
84. press enter
    
85. Now, we make the token for the web setup for kibana by typing:
    elasticsearch-create-enrollment-token –scope kibana
86. Press enter
87. It will give you a code to copy and paste back into kibana browser window:

88. Press: Configure Elastic
    It will then ask you for a verification code

89. Go back to the jail’s shell and copy/paste the command: kibana-verification-code
90. Press enter
91. Note the code to type in the box. Type in code and click: “Verify”

Wahoo!

You can now log in with the elastic account that you created/reset the password for way back when…

Let’s setup monitoring.

92. Click on the menu on the left – the three lines
93. Go to the bottom of the menu and click Stack Monitoring
94. Click: “Or, set up with self monitoring”
95. Click: “Turn on Monitoring”
96. Wait for it to finish

Make sure the everything is good!

Install FSCrawler and Tesseract OCR

97. type: pkg install wget
98. press enter
99. press “y”

The FSCrawler webiste is here

We need the location of the newest file: Located here. Note: Only the latest version has the ElasticSearch version 8 extensions. The other older versions will not work. You need the link of the latest one… at the bottom of the page… right click and copy link

100. Go back in the jail’s shell type this: cd
101. press enter
     ~you should be back at the root home folder~
102. Type: wget https://s01.oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-distribution/2.10-SNAPSHOT/fscrawler-distribution-2.10-20231023.160816-291.zip
103. press enter
104. it will download the files
105. type: unzip fscrawler-distribution-2.10-20231023.160816-291.zip
     This extracts the files
106. If you type ls and press enter. You should see the new files:

107. for ease of typing… rename the folder by typing this:
     mv fscrawler-distribution-2.10-SNAPSHOT fscrawler

Now… we initiate the first run of the fscrawler to create basic setting file for us to edit

108. type: ./fscrawler/bin/fscrawler scanfiles
109. press enter
110. type “y” to create
111. press enter
112. open the newly created settings file to edit by typing: nano .fscrawler/scanfiles/_settings.yaml
113. edit the url line to indicate the location of the files which are now appearing in /mnt
     url: “/mnt”
114. edit the elasticsearch: nodes: url: to show your jail’s ip address: 192.168.1.234
115. add a line and type: username: “elastic”
116. add a line and type: password: “bZb0v7BlO38sWyB0a1M7”
     use your password here from when you reset it earlier and wrote it down
117. we are not doing SSL verification
118. press control+o to save
119. press enter
120. press control+x to exit

121. Install tesseract OCR package by typing: pkg install tesseract
122. press enter
123. press “Y”
124. wait for it to finish
125. See if it all starts okay by typing the following: ./fscrawler/bin/fscrawler scanfiles

Set a Cron to Schedule the FSCrawler

126. type crontab -e
127. press enter
128. The screen enters a vi editor, press “i” to enter edit mode
129. set the runtime by using @600 for running every 10 minutes. Type: @600
130. then type the command /root/fscrawler/bin/fscrawler scanfiles
131. press esc
132. press :w
133. press enter
134. press :
135. press q
136. press enter

The search will run on the new schedule and create an index for you to search in Kibana

Fix the memory Issue

ElasticSearch, Kibana, and FScrawler are all running in Java virtual machines that will each will take up to 50% of the total system memory. I’m sure this made some sense to someone… THIS WILL CRASH your TrueNAS server. You need enough memory for ZFS to work, and if you accidentally decide to do anything, poof… system halt… process protection starts to kill off tasks that you might actually need.

Now we tweak the memory footprint for the JVM.

We are going to edit the system wide environment variables. In FreeBSD these are stored in the csh settings file

137. Type: nano /etc/csh.cshrc
138. We can add a line to set the JAVA_HOME. Type: setenv JAVA_HOME /usr/local/openjdk17
139. Add a line to fix the JAVA Memory. This has 2 parts, the starting (Xms) and max (Xmx)
     I’m picking approx 2 GB of memory use for each instance, that is 2 GB x 3 processes = 6 GB.
     Type: setenv FS_JAVA_OPTS “-Xmx2048m -Xms2048m”
140. ctrl + o
141. enter to save
142. ctrl +x
143. type: exit
144. press enter
145. You should be back at the jail’s control panel, press the “restart” button

DONE!

At this point, the jail system should reboot and start without any errors.
It will take a few moments for the servers to all start-up. Then, you can then go to Kibana dashboard and monitor the progress, do a search, look through your index.

The cron scheduler will start FSCrawler in a few minutes. It will first index the folders, and then start on the files. You may want to modify the settings in the fscrawler settings further to do OCR, not do OCR, exclude some files, etc.

Keep in mind… depending on how many files and the complexity, it might take days to complete.