Amit Nithian's Random Writings

Using JMX to modify the parameters of a running Solr instance

2011-09-26T21:49:00.000-07:00

Apache's Solr is a great software package to build your company's search engine around because of it's active community, great contributions, and solid foundations with Lucene at its core. If you are new to this package, then check out my earlier post on setting up Solr with Eclipse http://hokiesuns.blogspot.com/2010/01/setting-up-apache-solr-in-eclipse.html .

Normally, a company using Solr to power it's search engine will have a front end written in a web framework such as Ruby on Rails that makes calls to Solr through it's REST interface. One example is Zvents.com (my employer), where our front end is RoR that makes multiple calls to Solr to power various portions of the site. One common problem we encountered at Zvents was that a particular search would yield weird results and we'd execute the query passing the "&debugQuery=true" flag to study why that result appeared where it did. Afterwards; however, came the hard part.. tuning the ranking functions and refreshing the page to see if the results look better. There are two ways to do this:

Pass the modified set of ranking function parameters (bf, pf and/or qf) to Solr via the URL string
Modify solrconfig.xml and restart Solr to see what the results look like.

While #1 is certainly better than #2, both have their problems for the following reasons:

Restarting Solr can be slow and is not scalable. Each change requires a restart and this may take several minutes just to see the changes. Also it forces the person doing the tuning (say the search scientist/expert) to be able to understand the operational side (i.e. where Solr resides, how to restart it etc). This may be problematic for some people who are a averse to dealing with such operational details.
While passing the ranking function parameters via the URL works, it makes the URL significantly longer (which is a personal annoyance that most browsers have single line URL boxes) and this doesn't allow you to see the results of your change in the UI built by your front end team (without a special interface or requiring someone from that team to make changes to the UI code).

At Zvents, our initial solution to this involved building a special UI hosted within the Solr web application context, a JSP page + corresponding servlet with text boxes for each major ranking function parameter (qf, pf, bf) per handler. Our search experts would access this page and make the necessary changes which, upon submission of the form, would update the respective Solr handler's in-memory configuration map and thus the changes could be viewed in the UI built by our front end team. This was a great solution and certainly works; however, when Solr introduced JMX, it made more sense that such functionality live inside an MBean that exposed the ability to change these parameters using JConsole or any other UI with similar functionality.

Solr 2306- Modify default solrconfig parameters via JMX

When Solr added JMX support, it was a great way for administrators to be able to use a well known set of tools to understand what is going on inside of the JVM running Solr. Not only can you use JConsole to understand what the VM is doing but also what is going on inside of Solr.

This patch extends the built-in JMX support to also allow the default parameters (exposed via the <lst name="defaults">) to be exposed as modifiable attributes. When these parameters are modified, the changes are immediately noticed upon the issued search query without requiring any core reloads. To use this patch, follow the steps below and post any questions/comments on this JIRA issue https://issues.apache.org/jira/browse/SOLR-2306.

Step One: Checkout Solr, apply patch and build

svn co http://svn.apache.org/repos/asf/lucene/dev/trunk solr_trunk
cd solr_trunk
patch -p0 < <PATH_TO_PATH>/tuning_patch2.patch

cd solr
ant dist

Step Two: Setup and run example Solr instance

cd example
cp ../dist/apache-solr-4.0-SNAPSHOT.war webapps/solr.war
java -jar start.jar

Step Three: Load sample data

cd exampledocs
./post.sh *.xml

Step Four: Launch jconsole!

jconsole

In JConsole, choose the "MBeans" tab and along the left side, expand "/solr", "browse/", "org.apache.solr.component.SearchHandler", "Attributes". From here, you can select an attribute and change it's value.

Step Five: Modify parameters

For purposes of demoing the functionality of this patch, let's modify the "fl" parameter which describes the default set of field values to return. This can be overridden by passing &fl= on the URL; however, let's modify this parameter via JConsole to see it's effect. First though, let's look at some data by accessing http://localhost:8983/solr/browse

Notice that there is at least a name, price, and in-stock fields displayed. Using JConsole (make sure you followed the previous step), select the "fl" attribute and change it's value to only show "score,name"

and refresh your web browser pointing at the browse handler.

Notice that only the name is displayed along the top with the other fields that were initially present no longer there! Try this for other parameters and if you have JConsole pointing at your own search engine, try modifying the qf, pf, and bf parameters to see their effects. Of course keep in mind that to persist the parameter changes, make the appropriate changes in your solrconfig.xml file.

Conclusion

Through a fairly simple, contrived demo using the examples provided in the standard Solr distribution, you were able to change some default handler parameters for a particular search handler and immediately see its effects. Solr's JMX extensions provide a great way to peek inside of a running Solr instance without any major performance impacts. This patch takes this one step further and allows for real-time tuning of a running Solr instance. If you like this patch, and have login credentials on Apache's JIRA, please vote this up to help ensure that it gets rolled into a future release of Solr!

Using Hudson as a cron/workflow manager

2011-07-10T20:34:00.000-07:00

Every organization has scheduled jobs, whether it be daily database backups or hourly execution of log processing. Managing and running scheduled jobs is always a pain and the de-facto standard for scheduling such jobs is cron/anacron because it's easy and available on any Unix/Linux system. I'm sure many of you are familiar with the problems of using cron (lack of any job-flow control, logging, failure reporting etc) and having dealt with such problems at Zvents where I am a lead search/analytics engineer, I installed Hudson specifically as a cron server to solve these problems. For months prior, I used Hudson as a build server and it worked very well; however, it never crossed my mind to use Hudson as a cron server until I talked to someone else about this problem.

David Strauss wrote an interesting blog post (http://fourkitchens.com/blog/2010/05/09/drop-cron-use-hudson-instead) that I stumbled across when looking for materials for this post and it sums up what I was going to say beautifully so I won't repeat the major points here; however, there are a few other advantages to using Hudson as a cron which segues into the second part of my title, "workflow manager". To me, Hudson is a "framework" within which I can simply execute jobs and use the pre-built components necessary for things like archiving, notification on failure, concurrent execution etc. One feature that I don't think is talked about very much is the "workflow" like abilities you get by having jobs invoked when another job is complete.

As a user of Hadoop for processing large quantities of data, the community has been abuzz with workflow management tools and this has been a topic at various Hadoop meetups I have attended. People have often asked who uses the various workflow tools (Oozie vs Azkaban vs Cascading etc) and my somewhat abnormal suggestion has been to use Hudson to invoke scripts that wrap the invocation of Hadoop jobs. Hudson allows you to construct a simple workflow by invoking builds upon the completion of other builds. If you are looking for a non-transactional workflow manager to run your scheduled jobs, then consider using Hudson as it's battle tested and complete with features and a robust plugin architecture to develop your own add-ons in case one doesn't exist for your needs.

At Zvents, I have Hudson kick off an hourly log processing job for revenue recognition purposes which takes about 10-15 minutes to run. Upon completion, it kicks off a few other "builds" that operates on the results to produce some reports that aren't as critical should they take longer than an hour to complete. This allows the core revenue recognition jobs to always start and end within the hour and queue up the downstream jobs to complete when they do. While using Hudson as a workflow manager isn't as simple as say Azkaban, which uses configuration files to represent the dependencies between various jobs, it does the job fairly well for simple workflows. Also given that it's a build server, the integration with version control systems (SVN, GIT etc) makes it possible to be able to version important and have Hudson download from SVN the latest stuff before starting a build.

In short, Hudson is more than just a build system; it's a complete cron system with the abilities to setup simple workflows.

JDBC Connectivity to Hypertable!

2010-08-29T00:10:00.001-07:00

Hypertable (http://www.hypertable.org) is an implementation of the Google's BigTable which in short is a scalable, distributed sorted hashtable allowing for the storage of massive amounts of data across a cluster of commodity hardware. Hypertable has a rich query language (HQL) and a command line client akin to MySql's command line client; however, this can be limiting.

This article will cover two things:

1) JDBC Connectivity to Hypertable

2) Usage of this driver to graphically browse and modify a Hypertable instance

JDBC Driver

Hypertable uses the Thrift protocol to remotely connect and query an instance. This is a very useful method of reading/writing data but it's not standard and can be difficult to develop applications using Hypertable. For Ruby developers, there is a package called HyperRecord which provides an ActiveRecord like interface to Hypertable which makes integrating Hypertable into a Ruby (on Rails) application much easier. For Java, I recently released a JDBC driver that communicates through Thrift to Hypertable which now allows you to integrate Hypertable into a Java application via JDBC. One such application is integration with a graphical browser which was the motivation behind writing this driver.

Driver Limitations:

Only the latest timestamped version data is returned.
When retrieving metadata objects (ResultSet or Database), only the column families defined in the schema will be returned. In other words, column family qualifiers aren't going to be shown in the metadata objects although their value can be retrieved in code. Example: In code, you can do rs.getString("address:home") and rs.getString("address:work") while the metadata will only show "address" as a valid column name with no value unless 'address' explicitly has a value in the given table.

You can download the driver at http://github.com/downloads/ANithian/hyperjdbc/hypertable-jdbc_0.1.tar. Simply add all the jars to your classpath and use the driver "org.hokiesuns.hypertable.jdbc.HTDriver". An example URL is "jdbc:hypertable://192.168.116.128:38080" with no username, password or schema. An example application can be found by executing org.hokiesuns.hypertable.HypertableJDBCTester passing in the ThriftBroker hostname/ip as the command line argument.

Graphical Browser

The motivation behind writing this driver was to be able to view data in Hypertable using a graphical browser instead of the command line. There are numerous java based graphical browsers and the one that I tested with was SQLWorkBench/J (http://www.sql-workbench.net/). Rather than providing screenshots with explanations, I have created a video that should help show this driver in action and how easy it is to start talking to Hypertable using a third party graphical interface!

Conclusion

Along with integration for graphical browsing of Hypertable data, I believe that releasing a JDBC driver to Hypertable will help make its use more widespread and make it easier to plug this great technology into the vast Java application landscape. If you see any bugs, please file a bug report along with all the necessary information at http://github.com/anithian/hyperjdbc/issues

Setting up Apache Solr in Eclipse

2010-01-02T20:25:00.000-08:00

Apache's Solr is a powerful software package that allows you to develop your own search engine in no time. It's purely written in Java using Lucene at its core and can run inside any servlet container such as Tomcat (or Jetty). Eclipse is an IDE that makes developing Java applications incredibly easy because of its wealth of features such as code completion and refactoring capabilities not to mention the number of free plugins available to further make development easier. I find it much easier to keep everything contained in one place and being able to code, debug, and test inside of Eclipse makes developing my search engine much easier. This simple tutorial will show you how to setup Apache Solr to run inside Eclipse using a free third-party plugin that runs Jetty inside Eclipse.

You will need:

Eclipse (http://www.eclipse.org). I am using Eclipse 3.4 but this should also work just fine in Eclipse 3.5.
RunJettyRun plugin (http://code.google.com/p/run-jetty-run/wiki/GettingStarted)
Apache Solr (Any version should suffice. I am using 1.4.)

Step One: Basic Setup

Download and extract both Eclipse and Apache Solr tar files somewhere on your disk. Since Solr has some XML configuration files, I would also suggest installing the Eclipse WTP (Web Toolkit Platform) that gives you some good built-in XML editors.

Follow the Getting Started guide in the RunJettyRun wiki to install the plugin. It should be pretty fast and easy.

Step Two: Create your Java project

Create a standard Java project in Eclipse (File..New..Java Project). Call it what you wish (I called it "TestSolr"). The default options should be fine and click through the wizard to see all the options or click Finish at your first chance to get done faster.

Here you should see your TestProject in your workspace with a blank src folder.

Step Three: Setup the Solr webapp in your Eclipse project.

This is where the RunJettyRun plugin installed earlier gets used. This plugin allows you to develop, run and debug web applications inside Eclipse allowing you to take advantage of Eclipse's powerful code editing and debugging capabilities. It's also one of the simpler web application development plugins available. Eclipse supports full blown web development but for the purposes of this tutorial and developing needs, this plugin is more than enough.

Inside the TestSolr directory, create a folder called "webapp". Do this by right clicking on the "TestSolr" in the workspace, select "New" and then "Folder".
At a command prompt, unjar the apache-solr-1.4.0.war in this webapp folder. In Windows, this would be done by the following command:
jar -xvf c:\applications\apache-solr-1.4.0\dist\apache-solr-1.4.0.war
The contents of the war file should be in the webapp folder. To confirm, right click on the "TestSolr" folder in the Eclipse workspace and select "Refresh."
Add all the jar files in the webapp/WEB-INF/lib to the Build Path. This is done by selecting all the jars and right click on any of the jars (while all are selected), select the "Build Path" sub-menu and select "Add to Build Path"
Setup a solr home folder inside your project. For purposes of this tutorial, copy the "solr" folder from the "example/" folder in the directory containing solr into the "TestSolr" folder in your Eclipse workspace. By now, your "TestSolr" project layout should look something like this.

Step Four: Let's run this thing!

Now that you have setup your project, it's time to create a run configuration for Jetty and run this!

In Eclipse, go to the Run menu and select "Run Configurations...".
On the left rail, you should see "Jetty Webapp" as one of the run configuration types. Right click on this and select "New". The project should be "TestSolr" (if not, type in TestSolr in the text box labeled with "Project"). The name of the run configuration is populated with "TestSolr" and can be whatever you wish.
The default HTTP port is 8080 and can be left alone if you wish. To conform to Solr tutorials (making copying/pasting links easier, change this to 8983). Delete the HTTPs port 8443 since we aren't doing any SSL access. This will disable the SSL specific fields such as the keystore/password fields.
Change the "context" to "/solr" from "/". This is more for conformity with the solr tutorials in that all links in the wiki you to something like http://localhost:8983/solr/.
Type in "webapp" in the "WebApp dir" text box. This is the root of the web application and is the directory above the WEB-INF folder. This is where the solr war file was extracted and hence is the equivalent of deploying the solr war in your servlet container.

Your launch configuration should look like the following:

Now click the "Run" button which will save the changes and the console will hopefully start spewing out logs produced by both Jetty and Solr. Open a browser to http://localhost:8080/solr/admin/ and voila! you should see the Solr admin page. In scrolling through the console, you shouldn't see any exceptions thrown.

This simple tutorial just shows the basics of setting up a simple solr installation inside Eclipse using Jetty and running inside Eclipse. Notice that we don't have any of our own code running that may do some extra things (say custom analyisis, tokenizing etc). If we did have such code, then it would naturally lie in the "src/" folder; however, we need to instruct Eclipse to compile the code so that it goes in the WEB-INF/classes folder so it will run inside Jetty. To do this:

Right click on the TestSolr project and select "Properties".
Click on the "Java Build Path" on the left rail
Click on the "Source" tab and towards the bottom, change the default output folder to be "TestSolr/webapp/WEB-INF/classes".

Now you can develop your custom solr plugins directly in Eclipse and debug them immediately. Instead of running the TestSolr jetty instance, you can launch it in debug mode by going to the Run Menu and opening the "Debug Configurations.." sub-menu, selecting the Jetty configuration created earlier and click the "Debug" button. I would encourage familiarizing yourself with Eclipse's debugging capabilities for they are vast and amazing.

If you have any questions or comments, please post them and I'll do my best to answer them as quickly as possible.

Thinning down Aptana's RadRails to use with Eclipse

2009-10-24T01:21:00.000-07:00

I believe that Eclipse is the best damn IDE ever developed. The modular design (OSGI) coupled with its open plugin development environment and reasonable publication coverage makes it an ideal development environment. Companies have built development solutions on Eclipse technology and one such company is Aptana, the developer of RadRails, the development solution for web developers/Ruby on Rails developers.

A while ago, there was an Eclipse plugin called RDT (Ruby Development Toolkit) that provided "simple" Ruby support (a nice editor, some code completion etc.). Development of this plugin was discontinued and instead rolled into Aptana's large RadRails suite since RDT's developer was hired by Aptana. This obviously was great news for him but sad for me because I didn't want the heavy RadRails environment with its hundreds of plugin jars doing everything from HTML to Cloud Computing deployment. Given Eclipse's pluggable nature, I figured that I should be able to take a RadRails install and only extract the portions I wanted to give me my basic RDT back except now it's the latest and greatest.

Below is a directory listing showing the directories and jars to extract from the RadRails install to make your eclipse have Ruby development capabilities. I found this through trial and error and they seem to work just fine but if you find errors, post comments for others to see.

./eclipse/features

./eclipse/features/org.rubypeople.rdt_1.4.0.1254334821-7H7X7c7jrMElGv94Q

./eclipse/plugins

./eclipse/plugins/com.aptana.rdt.libraries.win32_1.0.1.jar

./eclipse/plugins/com.aptana.rdt.libraries_1.0.7.jar

./eclipse/plugins/com.aptana.rdt.rake_1.3.2.1254241359.jar

./eclipse/plugins/com.aptana.rdt.ui_1.3.2.1254332735.jar

./eclipse/plugins/com.aptana.rdt_1.3.1.1254334821.jar

./eclipse/plugins/org.epic.regexp_rdt_0.1.6.1252597203.jar

./eclipse/plugins/org.jruby_1.2.0.9419p2

./eclipse/plugins/org.kxml2_2.1.5.jar

./eclipse/plugins/org.rubypeople.rdt.branding_1.3.0.1254241359.jar

./eclipse/plugins/org.rubypeople.rdt.core_1.3.2.1254332672.jar

./eclipse/plugins/org.rubypeople.rdt.debug.core_1.3.2.1254241359.jar

./eclipse/plugins/org.rubypeople.rdt.debug.ui_1.3.2.1254334821.jar

./eclipse/plugins/org.rubypeople.rdt.launching_1.3.0.1254241359.jar

./eclipse/plugins/org.rubypeople.rdt.refactoring_1.2.2.1252686004.jar

./eclipse/plugins/org.rubypeople.rdt.testunit_1.2.4.1254241359.jar

./eclipse/plugins/org.rubypeople.rdt.ui_1.2.4.1254254467.jar

I highly recommend creating a separate extension location to store these plugins so future upgrades become easier since you know the features and plugins in this extension location are solely dedicated to RDT.

I am not a serious web developer requiring me to install the full Aptana suite, which significantly slowed down my other development in Eclipse, so this little trick has worked well for me. Now I have my RDT back without all the unnecessary plugins!

JRuby + Merb + Google AppEngine MemCache

2009-08-05T20:41:00.000-07:00

Searching far and wide for a cheap Java based web hosting provider, I became interested in learning about Google's AppEngine, which offers free space for web applications within reasonable access limits. I also thought that this would be a good opportunity to learn some Ruby (JRuby) to write a web application. In short, this post is about how to configure MERB running on JRuby/Google AppEngine to use Google's MemCache store to store session data via memcache instead of cookies.

I wanted to do this because I didn't want my web application data stored in a browser's cookie and so either I could use MemCache or the DataStore and I chose memcache. I assume that you have an AppEngine project setup with MERB as your web framework of choice.

All the changes described are located in /war/WEB-INF/config/init.rb

Load the APIs (place these near the top of init.rb)
# Load the apis
ENV['APPLICATION_ROOT'] = File.join(File.dirname(__FILE__), '..')
require 'appengine-apis/local_boot'
Change the session store from cookie to memcache
c[:session_store] = 'memcache'
Add the following to the Merb::BootLoader.after_app_loads section # This will get executed after your app's classes have been loaded.
Merb::MemcacheSession.store = AppEngine::Memcache.new
module AppEngine
# For the memcache-client gem.
class Memcache
include Merb::MemcacheStore
end
end

The first two and a good portion of the third step are standard Merb as far as I understand (although it took me some digging through the Merb code to find this). The second part of the third step is what I discovered when looking at the source code to see how the standard Memcache gem is integrated into the memcache store. Once I added this mixin AppEngine module, I had session data stored in memcache! Post comments if this works or doesn't work for you and/or if you have any other suggestions, please post so that others can find it!