SOLR Tracking for OOTB Support Tools
Some customers told me that last week I was a little bit eventually consistent, and this post is somehow related. I was at the Beecon 2017 in Zaragoza, the Alfresco Developer Conference, time to re-connect, re-charge, and re-commit with Alfresco Community. The event was really amazing (congrats to the organizers and those who made it happen, great event). I have been in previous Alfresco meetups, devcons and summits, but this was my first time in a Hackathon. At first I wasn’t sure about attending to Hackathon but Francesco Corti and Angel Borroy (MVP) wisely adviced me to go. I was not sure about the projects either, or how I could help. I use Support Tools a lot in our Alfresco projects, both original Enterprise Support Tools or OOTB Community version one. I knew the project features, and Support Tools saved my day more than once. I had some ideas in mind, but probably too far from my development skills. Anyway, I thought and proposed some of them, I do actually here:
- Hot edit mode for «System Information» parameters. This feature would kill Alfresco Community reboots – I often said that I could have studied a second master during years of Alfresco reboots. This will be a good idea for Enterprise Support Tools too (and in this case may be JMX based).
- A Libreoffice transformation process watchdog (another cause of Alfresco-based headaches is when Libreoffice hangs)
- An enhaced LDAP or user directory administration, for tasks like LDAP syncronizing or testing an LDAP configuration.
- A SOLR Tracking page with information about your indices.
Finally, I chose the last one, the SOLR Tracking page. Between the other groups in Hackathon, there was also very interesting projects such as Enhanced Rule Management for reusing static content rules in Data Dictionary, or User managed metadata extraction on Text PDF, with very good and promising demos. I started with the SOLR Tracking page (with the magic of Bindu Wavell and the potion of Axel Faust). By the way, Axel was with Cache resets, and Bindu et al. with log snapshots. All of this is a very helpful stuff for supporting projects. A final mention for Alfresco Audio Transcriber team, with Mikel Asla and Fernando Gonzalez, for enjoying a great time during the week.
Back to SOLR tracking feature page, the idea was to have some additional page in the Alfresco Administration console to help SOLR monitoring, including:
- Indexation progress and aproximate indexing times, to track indexation process.
- Disk indices size, and memory used by SOLR: Indices size is an important part to monitor. SOLR does merging continuosly and needs enough space for merging task. It’s recommended to have enough space for your indices in disk partition, with at least free space equal to your current indices size. Besides, it is interesting to know about the JVM Heap memory used by each store, to calculate JVM resources in SOLR nodes properly. In addition, there is a known formula to calculate needed JVM memory for SOLR instances.
- numDocs, maxDoc, deletedDocs values for each store: They give you an idea of the number of documents indexed and the fragmentation of your index. Take into consideration that a large number for deleted documents is not healthy for your Alfresco/SOLR system. Deleted documents tie up disk space in the index, cause they are marked as «deleted» but they are still there and they are part of the index. So they consume RAM resources, and mainly, search throughput is lower (much lower). A good idea is to monitor maxDoc/numDocs – 1.0 parameter, and if it is about 0.4 (for example), it may be a good idea to do a full reindex and optimization.
The trick for this new page is that there are two SOLR reports that helps obtaining all of this information (SUMMARY and STATUS reports). We could get them in a convenient JSON format from Javascript Console, and finally creating a webscript for the SOLR tracking page. Also a JSON webscript for consuming data via webscript from external tools such as Nagios/Icinga is available. I did not test it for SOLR 6 (only for SOLR 4 in Alfresco 201602GA). This is the probe of concept with the results in our corporate Community instance. The corresponding code is here.
Finally, I also did a script for Nagios/Icinga, for getting this SOLR info from the webscript. This is the result in Nagios:
Links:
- [1] https://community.alfresco.com/docs/DOC-6575-projects-and-teams-beecon-hackathon-2017
- [2] https://www.zylk.net/actualidad/how-to-track-solr-indexation-process-in-alfresco/
- [3] https://github.com/binduwavell/ootbee-support-tools/tree/solr-summary
- [4] http://docs.alfresco.com/5.2/concepts/solrnodes-memory.html
- [5] https://www.elastic.co/blog/lucenes-handling-of-deleted-documents
- [6] https://www.zylk.net/actualidad/performing-a-full-reindex-with-solr-for-alfresco-ecm/
