RkBlog

Advanced fulltext searching with Solr

Solr is a search server using Lucene to perform full text searching on indexed data. The most interesting part is that Solr uses a REST XML/JSON/HTTP API to add, modify, delete and search data so it's quite easy to use. However Solr and Lucene are written in Java so some Java servlet hosting is "required" :) If you need something bigger than for example Sphinx you can check this project out (Solr is used by Digg or sourceforge).

Enterprise search with PHP and Apache Solr
Search smarter with Apache Solr, Part 1: Essential features and the Solr schema
Search smarter with Apache Solr, Part 2: Solr for the enterprise
PDFs on Slideshare
A simple Python script for Solr would look like this:
from httplib import HTTPConnection

def add(item_id, title, description, text):
	"""
	Add a document to index
	"""
	DATA = '''<add><doc>
<field name="id">%d</field>
<field name="title">%s</field>
<field name="description">%s</field>
<field name="text">%s</field>
</doc></add>''' % (item_id, title, description, text)
	con = HTTPConnection('0.0.0.0:8983')
	con.putrequest('POST', '/solr/update/')
	con.putheader('content-length', str(len(DATA)))
	con.putheader('content-type', 'text/xml; charset=UTF-8')
	con.endheaders()
	con.send(DATA)
	r = con.getresponse()
	if str(r.status) == '200':
		print r.read()
	else:
		print r.status
		print r.read()

def commit():
	"""
	commit changes
	"""
	DATA = '<commit/>'
	con = HTTPConnection('0.0.0.0:8983')
	con.putrequest('POST', '/solr/update/')
	con.putheader('content-length', str(len(DATA)))
	con.putheader('content-type', 'text/xml; charset=UTF-8')
	con.endheaders()
	con.send(DATA)
	r = con.getresponse()
	if str(r.status) == '200':
		print r.read()
	else:
		print r.status
		print r.read()

def search(key):
	"""
	perform a search
	"""
	con = HTTPConnection('0.0.0.0:8983')
	con.putrequest('GET', '/solr/select?q=title:%s&start=0&rows=10&fl=id,title,description' % key)
	con.endheaders()
	con.send('')
	r = con.getresponse()
	if str(r.status) == '200':
		print r.read()
	else:
		print r.status
		print r.read()

add(5, 'Django search engine', 'blaaaa', 'text')
add(6, 'Google search', 'blaaa', 'text')
add(7, 'Solr fulltext search engine', 'blaaaa', 'text')
add(8, 'Car engine', 'blaaaa', 'text')
commit()
search('searching')
For a simple schema like this:
<fields>
  <field name="id" type="string" indexed="true" stored="true" required="true" />
  <field name="title" type="text" indexed="true" stored="true"/>
  <field name="description" type="text" indexed="true" stored="true"/>
  <field name="text" type="text" indexed="true" stored="true"/>
  </fields>
  <uniqueKey>id</uniqueKey>

› Read more

Diamanda Application Set 2008.07 Test 3 Released

Diamanda 2008 Test 3 release contains mainly fixes, and enchantments submited after Test 2. It's the last release not compatible with newforms-admin (and so compatible with older Django-SVN code) so for older projects you can use code from this release. Test 4 will be NFA compatible, and probably hit normal stable release with Django 1.0.

All the features of Diamanda 2008 you can check on this site, which runs on Diamanda 2008 Test 3. I'm updating the documentation from 2007 to 2008 release. Comments, and showcase links are welcomed :)

Diamanda 2008 main features:

  • Much improved forum (discussion board) – MyghtyBoard
  • OpenID support
  • Cleaner code
  • Compatibility with the "latest" Django code
  • Download ZIP or download tar.bz2

› Read more

Diamanda 2008 Test 2 coming soon

Diamanda 2008.05 Test 2 will go public in about a week. Changes planned/implemented since 2008.04 Test 1:

  • Removed less important apps (will be available in diamanda-extras)
  • Forum, and overall UI improved
  • Support for new unread posts and topics, and links to first unread post
  • markItUp! BBCode on the forums, and Markdown on the wiki-content module
  • Wiki-almost-like content module with Markdown syntax and CBC tags
After Test 2 there will be one more "test" release – 2008.06 Test 3 with fixes, clean-ups, and - data importers (move data from other scripts, for example PHP forums and CMS to Diamanda). After a period of testing a stable 2008 release will be available.

› Read more

Diamanda 2008.07 Test 2 Released

You can download second Test/Beta release of Diamanda 2008. Compared to Test 1 there is a lot of changes in the code.

  • Prefix, moderators and anonymous support in forums. Marking topics as solved.
  • Main Page (and RSS) feed of new and updated entries (pages and posts by default)
  • OpenID support (you can login by openID, then do a quick registration with no passwords. OpenID user will be auto-logged on his Django account.)
  • Few minor changes in all components
In the next 1-2 weeks Test 3 will be released, and after a week a Final release of Diamanda 2008 will be made. Test 3 will contain fixes and few unfinished features (polishing OpenID, content submission, text captcha)

› Read more

Diamanda 2008.04 Test1 released

I've released a snapshot/beta package of current Diamanda code. This release is designed for (latest) Django-SVN. Bigger changes are: rewritten userpanel (uses Django auth views), migration to newforms in forum (MyghtyBoard) and boxcomments, UI fixes and enchantments. I have some more features on the ToDo list, and I'm planning one or two Test releases before stable one. You can download tar.bz2 from code.google.com, or use SVN:

svn checkout http://diamanda.googlecode.com/svn/trunk/ diamanda-read-only

› Read more

AppEngine everywhere :)

Yes, everyone is talking about it :) It looks cool, but I have to wait in line... But here I have a quote from one of "reviews":

For virtualization reasons only the pure Python is supported and a subset of the Python standard library.
Hm... so I can't have PIL there? If so web based services and tools will get big attention (google charts and not matplotlib, pixenate and others and not PIL).
disk storage is up to 500M, both incoming and outgoing traffic is restricted to 10G per day each, up to 2000 emails per day, and up to 200 million megacycles of CPU per day
All my sites would fit :)

› Read more

RkBlog main page Search RSS Contact