Was translating a comment in the Japanese SEN tokenization library.
It seems like if your text includes the Unicode right arrow character, Google somehow gets confused about the encoding. Saw this on both Firefox and Safari. Not a big deal, strangely comforting to see even the big guys trip up on character encodings.
OK: サセン
OK: チャセ
Not OK: サセン→チャセ?
It's funny how certain facts are "obvious" to some folks, stuff they've known a long time, but come as a pleasant surprise to others. Chances are you know at least half of these, but no harm in double checking!
If you do a standard checkout of the Lucene/Solr codebbase you also get a dev-tools directory. One interesting tidbit in there is an Excel spreadsheet for estimating the RAM and disk requirements for a given set of data. Be sure to notice the tabs along the bottom, tab 2 is for memory/RAM estimates, and tab 3 is for disk space.
Full URL: http://svn.apache.org/repos/asf/lucene/dev/trunk/dev-tools/size-estimator-lucene-solr.xls
You may have heard that the recent release of Java 7 has what sounds to me like some serious problems which are discussed on the Lucid Imagination blog. The most telling line i found there -
"These problems were detected only 5 days before the official Java 7 release,
so Oracle had no time to fix those bugs, affecting also many more
applications."
Granted, this is from Uwe Schindler as quoted on Lucid's site - not directly from Oracle. But I have to wonder about any product that is released with known serious flaws like this when they ONLY had five days' notice. I've seen software halted hours before its intended release to investigate a potentially serious bug; did Oracle have to meet revenue at end of quarter, and 'damn the torpedoes'?
I know Oracle is not (the old) Hewlett Packard which had a legendary commitment to quality. When bugs were found in the earliest HP 3000, Dave Packard made his sales rep buy them back from customers - even from those customers who were happy with their purchase. The reason: The system did not perform as advertised. If every one of your users is going to be impacted, some in very subtle ways that may produce incorrect results - wouldn't you agree that five days is enough time to stop the presses?
Tell me what you think...
By default search engines sort results by relevance or "score", to try and bring the best match to the top of the results list. That's normally what users want, but occasionally you might want to sort by a different field, such as date, title or author. Lucene and Solr support this in various ways, as do many other search engines.
When it comes to sorting by titles or author names, most languages sort words with similar rules, and this is the character ordering that's built into Unicode. But a few languages are different, they may have different policies on accented characters, for example. Java includes to concept of "locale" to represent some language differences, such as currency and date formats, and it can also encode these differences in preferred order. However, apparently the performance isn't great, so sorting in some languages can be slow, or there may not be a locale for a specific language/dialect.
Lucene does include an alternate "collator" class that claims to fix this. It allows for non-default Unicode sorting rules, without the slowdown normally associated with locales. The doc mentions Farsi, Danish and Turkish as examples. Although I haven't tried it, since it's buried a bit in the code tree, I wanted to surface it in a post.
The top URL (in case formatting gets lost) is:
Usage scenarios are given in package.html
SMILA (SeMantic Information Logistics Architecture) is a Eclipse project that provides an extensible framework for building search applications to access unstructured information in the enterprise. It provides a integrated package based on Lucene that includes crawlers, connectors and the interfaces needed to manage it using existing infrastructure. The main goal of SMILA is to reduce the risk of investment and IT costs by providing a common development framework that can be used to build semantic applications and by standardizing a lot of the code.
SMILA attempts to provide economies of scale while providing the option to use highly specialized solutions or plug-ins as needed. It also provides the opportunity for a company to reuse interfaces from internal projects that use Lucene.
The first fully tested (to make certain there are no legal issues due to third party code) official release is available. Version 0.7 also adds Web Service API support and Solr integration (access to Apache Solr REST API).
SMILA has been getting more German press (it was created by Empolis GmbH and Brox IT Solutions GmbH) in the last year but very little in this country. The last I spotted was as part of a 25 minute talk on Searching the Cloud - the EclipseRT Umbrella! at EclipseCon 2010 in March.
Version 0.9 is scheduled for November 30, 2010 and is supposed to include some more third party components (that have completed the IP process). It will be interesting to see if some of those components are from American companies and if they find a way to build bridges to other Eclipse projects that use semantic technologies. I found some newsgroup posts last year about creating a new Eclipse project to do that but nothing seems to have happened.
GitHub has a Chansonnier project based on SMILA, but its part of the authors bachelor's degree thesis project. It is a search application that indexes songs imported from the web, with parameters like language and emotion. Its useful as a sample SMILA application that isn't part of the official distribution. The SMILA project has a lot of potential but hasn't found a way to appeal to a wider audience yet
Google has announced its Google I/O 2010 to be held in San Francisco May 19-20 at the Moscone Center.
I think this is their third such annual event, and it's always been a full two days of information. The good news is the price is $400 per person (until April 15), a bargain really. The bad news? You'll need to bring four or five people from your company to hit all of the sessions in each track!
This conference is VERY technical, VERY good. You get the most from it if you are a developer, you know Java, Ajax, Python, or the other technologies Google uses in its various products. You won't find much in the way of marketing fluff here: in our experience, most presenters are Google developers.
The conference is being held the same week that Gilbane content management conference comes back to San Francisco. Bad timing for them, but good for you: you can probably walk to the nearby Westin at lunch and maybe catch the exhibits.
Last year, attendees received a free phone for development purposes on the Android OpSys; who knows what they might give away this year - besides the expected cool T-shirt!
Register at http://code.google.com/events/io/2010/.
"The method getTextContent() is undefined for the type Node"
You get 3 of these, in the source files ReutersService.java and TestConfig.java
A Web fix that doesn't work:
You'll see suggestions that org.w3c.dom.Node.getTextContent() is only available as of Java 1.5. But when you check you see you ARE running with Java 1.5 or later.
You can quickly check this by right clicking on the project, Properties -> Java Compiler, and confirm that 1.5 or above are in the drop down lists.
The order of the classpath needs to be tweaked in Eclipse project; shove the xml-apis-1.0.b2.jar all the way to the bottom, past the built in JVM libraries.
For more details, and how you would know this, read on!
