Nabble - Apache Tika - Development

Re: Tika report due October 8th

2008-10-08T16:16:50Z

Hi,

On Thu, Oct 2, 2008 at 11:17 PM, Bertrand Delacretaz
<bdelacretaz@...> wrote:
> Find a draft below - I'll be offline next week, feel free to finalize and post.

Posted, thanks!

BR,

Jukka Zitting

[jira] Updated: (TIKA-164) Update nekohtml version

2008-10-07T02:41:44Z

[ https://issues.apache.org/jira/browse/TIKA-164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting updated TIKA-164:
-------------------------------

Component/s: parser
Affects Version/s: (was: 0.1-incubating)
Issue Type: Improvement (was: Task)

+1

For the record, the nekohtml change history is at http://nekohtml.sourceforge.net/changes.html

> Update nekohtml version
> -----------------------
>
> Key: TIKA-164
> URL: https://issues.apache.org/jira/browse/TIKA-164
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Sami Siren
>
> Latest version currently available is 1.9.9.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-165) update icu4j

2008-10-07T02:37:44Z

[ https://issues.apache.org/jira/browse/TIKA-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting updated TIKA-165:
-------------------------------

Component/s: parser
Issue Type: Improvement (was: Task)

+1

> update icu4j
> ------------
>
> Key: TIKA-165
> URL: https://issues.apache.org/jira/browse/TIKA-165
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Affects Versions: 0.1-incubating
> Reporter: Sami Siren
>
> Latest version currently available is 3.8.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-167) Tika presentation @ ApacheConUs 2008: review

2008-10-07T00:30:44Z

[ https://issues.apache.org/jira/browse/TIKA-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637401#action_12637401 ]

Jerome Charron commented on TIKA-167:
-------------------------------------

Very nice presentation.
Just a detail : Tika doesn't come from my son's name, but from the name my son gives to its "doudou" (I don't know what is the english for "doudou" ... do you see what I mean?)

> Tika presentation @ ApacheConUs 2008: review
> --------------------------------------------
>
> Key: TIKA-167
> URL: https://issues.apache.org/jira/browse/TIKA-167
> Project: Tika
> Issue Type: Task
> Components: documentation
> Affects Versions: 0.2-incubating
> Reporter: Paolo Mottadelli
> Attachments: ApacheConUS2008_Tika_PaoloMottadelli.pdf
>
>
> As I have not been involved in the development process, it would be great if someone could review the Tika part of my presentation. I am attaching a rough version of my slides concerning the Tika presentation and listing some *** Open Points ***. Please, let me know if I am out of scope in some parts and if I can get better anyhow.
> *** Open Points: ***
> * What does TIKA mean? (literally)
> * How many registered media types; glob/magic header (slide 7)
> * How many supported media types (slide 7)
> * How many committers? (slide 7)
> * Do we have a download/chackout history for Tika? (eventually slide 10)
> * Future goals; to be completed? (slide 31)
> * Next parsers to be implemented? (slide 32)
> * Who uses Tika? projects using Tika (slide 33)
> * Integration scenarios with other Lucene projects (slide 34)
> * Related projects: others? (slide 34)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-167) Tika presentation @ ApacheConUs 2008: review

2008-10-06T16:42:44Z

[ https://issues.apache.org/jira/browse/TIKA-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637302#action_12637302 ]

Jukka Zitting commented on TIKA-167:
------------------------------------

The presentation looks great!

> What does TIKA mean? (literally)

"Tika" comes from the name of Jérôme Charron's (who first proposed the project in 2006) son.

> How many registered media types; glob/magic header (slide 7)

We currently have 77 registered mime types (plus 36 aliases), 179 glob patterns, and 18 magic patterns.

> How many supported media types (slide 7)

Depends on what you mean by "supported". We currently have 15 parser classes configured for a total of 66 mime types.

> How many committers? (slide 7)

Six, plus Dave Meikle who was just voted in.

> Do we have a download/chackout history for Tika? (eventually slide 10)

Not really. I could try to dig something up if you want, though I expect the numbers to be fairly low still as we've kept a relatively low profile so far.

> Future goals; to be completed? (slide 31)

Main goals off the top of my head, see the issue tracker for more:

- Improved metadata handling, perhaps with XMP support
- Better configurability of Tika
- Improved media type registry
- More parser implementations

> Next parsers to be implemented? (slide 32)

- Office Open XML based on a POI upgrade
- Structural parsers (i.e. more than just a flat text stream) for PDF, Word, OpenDocument, etc.
- More multimedia formats: image, audio, video

> Who uses Tika? projects using Tika (slide 33)
> Integration scenarios with other Lucene projects (slide 34)

Not that many now that we're still incubating. Beyond Nutch we have at least Apache Jackrabbit with a sandbox component with Tika support, the Droids lab (to be incubated) that is currently adding Tika integration, and the UIMA project (incubating) that has a proposed patch with Tika support.

> Related projects: others? (slide 34)

Aperture (http://aperture.sourceforge.net/) is another project with similar (though wider) goals.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Tika report due October 8th

2008-10-02T14:17:23Z

Hi,

See http://wiki.apache.org/incubator/October2008

Find a draft below - I'll be offline next week, feel free to finalize and post.

-Bertrand

Apache Tika is a toolkit for detecting and extracting metadata and
structured text content from various documents using existing parser
libraries. Tika entered incubation on March 22nd, 2007.

Community
Dave Meikle was just voted in as a new committer. Paolo Mottadelli
will present Tika at ApacheCon US.

Development
Tika 0.2 should be released soon.
Usage documentation has been added to the website.

Issues before graduation:
The current plan is to graduate as a Lucene subproject, which could
happen soon as the incubation criteria seem to be met.

Re: New Tika committer

2008-10-01T23:20:56Z

On Wed, Oct 1, 2008 at 10:50 PM, Jukka Zitting <jukka.zitting@...> wrote:
> ...Dave, welcome to the team!..

Congrats Dave, and welcome!
-Bertrand

New Tika committer

2008-10-01T13:50:13Z

Hi,

I'm proud to announce that Dave Meikle will be joining the Tika team
as a committer. Based on Dave's enthusiasm and the high quality of his
contributions the Tika PPMC recently decided to offer him
committership in the project. The Incubator PMC has approved the
decision and Dave has accepted the nomination, so as soon as we get
the administrative bits in order he will be a full member of the Tika
team.

Dave, welcome to the team!

BR,

Jukka Zitting

Re: Apache Tika on the Fast Feather Track

2008-09-30T14:58:24Z

Would love to have Tika people at the Search BOF, too, if you feel it
is the right fit. It would be good to have someone update the search
community on Tika's progress. See http://wiki.apache.org/apachecon/BirdsOfaFeatherUs08

-Grant

On Sep 28, 2008, at 6:56 AM, Jukka Zitting wrote:

> Hi,
>
> On Sun, Sep 28, 2008 at 1:21 AM, Jukka Zitting <jukka.zitting@...
> > wrote:
>> ApacheCon US is coming up and there will again be a Fast Feather
>> Track
>> where we could present the latest developments in Tika. We have
>> implemented quite a lot of new stuff since the last ApacheCon, and it
>> would be good to showcase our progress.
>
> I just noticed (thanks to Grant's promo message) that Paolo Mottadelli
> is already doing a full session [1] on Tika, so an extra FFT talk
> might not be needed.
>
> [1] http://us.apachecon.com/c/acus2008/sessions/12
>
> BR,
>
> Jukka Zitting

Re: Apache Tika on the Fast Feather Track

2008-09-30T08:27:01Z

Hi,

> I just noticed (thanks to Grant's promo message) that Paolo Mottadelli
> is already doing a full session [1] on Tika, so an extra FFT talk
> might not be needed.
> [1] http://us.apachecon.com/c/acus2008/sessions/12

I have been working with Tika since last spring, using Tika for
building the Open XML support in Alfresco ECM. I used Tika as a user,
more than a Tika developer; anyway, I am really proud to present Tika
at ApacheConUS.

I am completing a first version of the Office Open XML format Tika
Parser, and I am looking forward to contribute it to the project. At
the moment some *** POI(3.5-beta3)_dependencies -> maven *** related
issues (and no time to solve them) are making me waiting.

As I have not been involved in the development process, it would be
great if someone could review the Tika part of my presentation.
I also listed some Open Points that would be great to 'close'.

I have created an Jira issue on this, attaching thta part of my presentation:
https://issues.apache.org/jira/browse/TIKA-167

Thank you very much for your so valuable contribution.

Cheers,

--
Paolo Mottadelli: http://www.paolomottadelli.com
Sourcesense - making sense of Open Source: http://www.sourcesense.com

[jira] Updated: (TIKA-167) Tika presentation @ ApacheConUs 2008: review

2008-09-30T08:19:44Z

[ https://issues.apache.org/jira/browse/TIKA-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paolo Mottadelli updated TIKA-167:
----------------------------------

Attachment: ApacheConUS2008_Tika_PaoloMottadelli.pdf

Attaching a PDF version (limited to the Tika part) of the presentation

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-167) Tika presentation @ ApacheConUs 2008: review

2008-09-30T08:19:44Z

Tika presentation @ ApacheConUs 2008: review
--------------------------------------------

Key: TIKA-167
URL: https://issues.apache.org/jira/browse/TIKA-167
Project: Tika
Issue Type: Task
Components: documentation
Affects Versions: 0.2-incubating
Reporter: Paolo Mottadelli
Attachments: ApacheConUS2008_Tika_PaoloMottadelli.pdf

As I have not been involved in the development process, it would be great if someone could review the Tika part of my presentation. I am attaching a rough version of my slides concerning the Tika presentation and listing some *** Open Points ***. Please, let me know if I am out of scope in some parts and if I can get better anyhow.

*** Open Points: ***
* What does TIKA mean? (literally)
* How many registered media types; glob/magic header (slide 7)
* How many supported media types (slide 7)
* How many committers? (slide 7)
* Do we have a download/chackout history for Tika? (eventually slide 10)
* Future goals; to be completed? (slide 31)
* Next parsers to be implemented? (slide 32)
* Who uses Tika? projects using Tika (slide 33)
* Integration scenarios with other Lucene projects (slide 34)
* Related projects: others? (slide 34)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-166) Update HTMLParser to parse contents of meta tags

2008-09-29T12:08:44Z

[ https://issues.apache.org/jira/browse/TIKA-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dave Meikle updated TIKA-166:
-----------------------------

Attachment: TIKA-166.diff

Patch to implement the functionality. This can be used for further discussion as I am not sure if we want to include the http-equiv info as well.

> Update HTMLParser to parse contents of meta tags
> ------------------------------------------------
>
> Key: TIKA-166
> URL: https://issues.apache.org/jira/browse/TIKA-166
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Affects Versions: 0.1-incubating
> Reporter: Dave Meikle
> Priority: Minor
> Attachments: TIKA-166.diff
>
>
> Enhancement of the HTMLParser to return the HTML <meta> tags found in the document in the Metadata object, as raised in the mailing list.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-166) Update HTMLParser to parse contents of meta tags

2008-09-29T12:04:44Z

Update HTMLParser to parse contents of meta tags
------------------------------------------------

Key: TIKA-166
URL: https://issues.apache.org/jira/browse/TIKA-166
Project: Tika
Issue Type: Improvement
Components: parser
Affects Versions: 0.1-incubating
Reporter: Dave Meikle
Priority: Minor

Enhancement of the HTMLParser to return the HTML <meta> tags found in the document in the Metadata object, as raised in the mailing list.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-165) update icu4j

2008-09-29T10:26:44Z

update icu4j
------------

Key: TIKA-165
URL: https://issues.apache.org/jira/browse/TIKA-165
Project: Tika
Issue Type: Task
Affects Versions: 0.1-incubating
Reporter: Sami Siren

Latest version currently available is 3.8.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-164) Update nekohtml version

2008-09-29T10:22:45Z

Update nekohtml version
-----------------------

Key: TIKA-164
URL: https://issues.apache.org/jira/browse/TIKA-164
Project: Tika
Issue Type: Task
Affects Versions: 0.1-incubating
Reporter: Sami Siren

Latest version currently available is 1.9.9.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Planning Tika 0.2

2008-09-28T15:43:28Z

2008/9/28 Sami Siren <ssiren@...>

> Jukka Zitting wrote:
>
>> I think the current trunk is good enough to be released.
>>
>>
> +1
>
> --
> Sami Siren
>
>

If it mattered from me I would give it a +1, but since it doesn't I will
just give it a smile :-)

Re: Planning Tika 0.2

2008-09-28T10:18:19Z

Jukka Zitting wrote:
> I think the current trunk is good enough to be released.
>
+1

--
Sami Siren

Re: Apache Tika on the Fast Feather Track

2008-09-28T03:56:08Z

Hi,

On Sun, Sep 28, 2008 at 1:21 AM, Jukka Zitting <jukka.zitting@...> wrote:
> ApacheCon US is coming up and there will again be a Fast Feather Track
> where we could present the latest developments in Tika. We have
> implemented quite a lot of new stuff since the last ApacheCon, and it
> would be good to showcase our progress.

I just noticed (thanks to Grant's promo message) that Paolo Mottadelli
is already doing a full session [1] on Tika, so an extra FFT talk
might not be needed.

[1] http://us.apachecon.com/c/acus2008/sessions/12

BR,

Jukka Zitting

Apache Tika on the Fast Feather Track

2008-09-27T16:21:56Z

Hi,

ApacheCon US is coming up and there will again be a Fast Feather Track
where we could present the latest developments in Tika. We have
implemented quite a lot of new stuff since the last ApacheCon, and it
would be good to showcase our progress.

I'll be attending the ApacheCon and would be interested in presenting
Tika there. Anyone else interested?

BR,

Jukka Zitting

Re: Planning Tika 0.2

2008-09-27T15:46:10Z

Hi,

The following issues were remaining on the 0.2 roadmap:

TIKA-50 Unit tests are incomplete.
TIKA-61 Add namespaces to our metadata keys
TIKA-69 ParseUtils methods need to support Metadata
TIKA-74 Test Resources should be loaded by the class loader ...
TIKA-79 Mime type detection from file header appears to be failing
TIKA-80 Utility method in MimeUtils to perform full mime resolution ...
TIKA-121 MimeType.clean method no longer exists as a capability

None of them looked terribly urgent or blocking, so I just removed
them from the 0.2 roadmap.

I think the current trunk is good enough to be released.

BR,

Jukka Zitting

[jira] Updated: (TIKA-135) The command line files (tika.bat, tika.sh) are not usable

2008-09-27T15:35:44Z

[ https://issues.apache.org/jira/browse/TIKA-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting updated TIKA-135:
-------------------------------

Fix Version/s: (was: 0.2-incubating)

> The command line files (tika.bat, tika.sh) are not usable
> ---------------------------------------------------------
>
> Key: TIKA-135
> URL: https://issues.apache.org/jira/browse/TIKA-135
> Project: Tika
> Issue Type: Bug
> Components: cli
> Affects Versions: 0.2-incubating
> Environment: Windows XP; Java 1.5
> Reporter: Karl Heinz Marbaise
> Assignee: Chris A. Mattmann
> Attachments: batsh-patch1.diff
>
> Original Estimate: 0.08h
> Remaining Estimate: 0.08h
>
> The bat/sh file do not include the correct path to the jar's nor to the classpath where all other files can be found.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

ApacheCon US promo

2008-09-26T11:53:59Z

Cross-posting...

Just wanted to let everyone know that there will be a number of Lucene/
Solr/Mahout/Tika related talks, training sessions, and Birds of a
Feather (BOF) gatherings at ApacheCon New Orleans this fall.

Details:
When: November 3-7
Where: Sheraton, New Orleans, USA
URL: http://us.apachecon.com/c/acus2008/

Lucene:

Advanced Indexing Techniques by Michael Busch: http://us.apachecon.com/c/acus2008/sessions/7

Lucene Boot Camp (2 day hands-on training by me): http://us.apachecon.com/c/acus2008/sessions/69

Solr:

Solr out of the Box by Chris Hostetter: http://us.apachecon.com/c/acus2008/sessions/9

Beyond the Box by Hoss: http://us.apachecon.com/c/acus2008/sessions/10

Solr Boot Camp (1 day hands-on training by Erik Hatcher): http://us.apachecon.com/c/acus2008/sessions/91

Mahout:

Intro to Mahout and Machine Learning (by me): http://us.apachecon.com/c/acus2008/sessions/11

Tika:

Content Analysis for ECM with Apache Tika by Paolo Mottadelli : http://us.apachecon.com/c/acus2008/sessions/12

There's also one more Lucene session that is TBD, but it will be on
that same Wednesday as everything else. Chances are it will be an
intro to Lucene type talk.

BOFs: http://wiki.apache.org/apachecon/BirdsOfaFeatherUs08

Cheers,
Grant

ANNOUNCE: Application Period Opens for Travel Assistance to ApacheCon US 2008

2008-09-26T10:25:03Z

NOTE: This is a cross posted announcement to all Lucene sub-projects,
please confine any replies to general@lucene.

-------------

The Travel Assistance Committee is taking in applications for those wanting
to attend ApacheCon US 2008 between the 3rd and 7th November 2008 in New
Orleans.

The Travel Assistance Committee is looking for people who would like to be
able to attend ApacheCon US 2008 who need some financial support in order to
get there. There are VERY few places available and the criteria is high,
that aside applications are open to all open source developers who feel that
their attendance would benefit themselves, their project(s), the ASF and
open source in general.

Financial assistance is available for flights, accomodation and entrance
fees either in full or in part, depending on circumstances. It is intended
that all our ApacheCon events are covered, so it may be prudent for those in
Europe and or Asia to wait until an event closer to them comes up - you are
all welcome to apply for ApacheCon US of course, but there must be
compelling reasons for you to attend an event further away that your home
location for your application to be considered above those closer to the
event location.

More information can be found on the main Apache website at
http://www.apache.org/travel/index.html - where you will also find a link to
the application form and details for submitting.

Time is very tight for this event, so applications are open now and will end
on the 2nd October 2008 - to give enough time for travel arrangements to be
made.

Good luck to all those that will apply.

Regards,

The Travel Assistance Committee

RE: HTML tags

2008-09-24T19:01:10Z

Dave,

I thought it might have been my configuration at work so I tried again at
home on a clean machine tonight and I get the same error. My files are
identical to your diff. For me it won't enter the startElement() method for
the meta handler.

I attached the three files (HTMLParser, HTMLParserTester, testHTML.html).

I'm sure this is something simple. Maybe some kind of configuration
difference? I've tried Java 5 and 6 with no change. Do you have a zip of
all your dependent tika .jar files I can try to use? That is my only guess
now.

--Brian

-----Original Message-----
From: Dave Meikle [mailto:loompa@...]
Sent: Wednesday, September 24, 2008 5:04 PM
To: tika-dev@...
Subject: Re: HTML <meta> tags

Hi

2008/9/24 Brian Levay <brian.levay@...>

meta));
>
> The <meta> handler isn't being called. If I use /HTML/HEAD//node() the
> handler will get called for the <head> and <title> tags but it will skip
> right past the <meta> tags. I know the tika code is seeing the META tags
> because I see the tags trying to be matched in the startElement method of
> MatchingContentHandler. Any ideas?
>
> --Brian
>

I am using effectively the same thing in a local copy and have just re-based
it again HEAD (shown in the diff below), and it appears to be working fine
for me.

What is your test XML like?

Cheers,
Dave

Index: src/main/java/org/apache/tika/parser/html/HtmlParser.java
===================================================================
--- src/main/java/org/apache/tika/parser/html/HtmlParser.java (revision
698705)
+++ src/main/java/org/apache/tika/parser/html/HtmlParser.java (working
copy)
@@ -95,9 +95,11 @@
XPathParser xpath = new XPathParser(null, "");
Matcher body = xpath.parse("/HTML/BODY//node()");
Matcher title = xpath.parse("/HTML/HEAD/TITLE//node()");
+ Matcher meta = xpath.parse("/HTML/HEAD/META//node()");
handler = new TeeContentHandler(
new MatchingContentHandler(getBodyHandler(xhtml), body),
- new MatchingContentHandler(getTitleHandler(metadata),
title));
+ new MatchingContentHandler(getTitleHandler(metadata),
title),
+ new MatchingContentHandler(getMetaHandler(metadata),
meta));

// Parse the HTML document
xhtml.startDocument();
@@ -116,6 +118,17 @@
};
}

+ private ContentHandler getMetaHandler(final Metadata metadata) {
+ return new WriteOutContentHandler() {
+ @Override
+ public void startElement(
+ String uri, String local, String name, Attributes atts)
+ throws SAXException {
+ metadata.set(atts.getValue(0), atts.getValue(1));
+ }
+ };
+ }
+
private ContentHandler getBodyHandler(final XHTMLContentHandler xhtml)
{
return new TextContentHandler(xhtml) {

Re: HTML tags

2008-09-24T14:14:35Z

Hi Jukka,

2008/9/24 Jukka Zitting <jukka.zitting@...>

>
> The XHTMLDowngradeHandler wrapper will uppercase all element names and
> drop all namespaces and namespaced attributes, but as far as I can
> tell your code should still match the META tags. But there might be
> some bug in the XHTMLDowngradeHandler code that breaks things.
>

This is working fine for me with the XHTMLDowngradeHandler in the code. I
just sync'd with SVN before I posted.

Cheers,
Dave

Re: HTML tags

2008-09-24T14:04:13Z

Hi,

On Wed, Sep 24, 2008 at 10:17 PM, Brian Levay <brian.levay@...> wrote:
> I'll submit the updates when I'm done (along with unit tests). I'm having a
> problem though. I sync'ed my tika baseline this morning and the Matcher
> stopped matching the <meta> tags. Any idea what my be causing this?

Most likely the TIKA-140 fix that I committed recently. You may want
to try reverting this change:

--- incubator/tika/trunk/src/main/java/org/apache/tika/parser/html/HtmlParser.java 2008/09/22
22:57:10 698027
+++ incubator/tika/trunk/src/main/java/org/apache/tika/parser/html/HtmlParser.java 2008/09/22
23:00:27 698028
@@ -102,7 +102,7 @@
// Parse the HTML document
xhtml.startDocument();
SAXParser parser = new SAXParser();
- parser.setContentHandler(handler);
+ parser.setContentHandler(new XHTMLDowngradeHandler(handler));
parser.parse(new InputSource(Utils.getUTF8Reader(stream, metadata)));
xhtml.endDocument();
}

> The <meta> handler isn't being called. If I use /HTML/HEAD//node() the
> handler will get called for the <head> and <title> tags but it will skip
> right past the <meta> tags. I know the tika code is seeing the META tags
> because I see the tags trying to be matched in the startElement method of
> MatchingContentHandler. Any ideas?

The XHTMLDowngradeHandler wrapper will uppercase all element names and
drop all namespaces and namespaced attributes, but as far as I can
tell your code should still match the META tags. But there might be
some bug in the XHTMLDowngradeHandler code that breaks things.

BR,

Jukka Zitting

Re: HTML tags

2008-09-24T14:03:30Z

Hi

2008/9/24 Brian Levay <brian.levay@...>

> I'll submit the updates when I'm done (along with unit tests). I'm having
> a
> problem though. I sync'ed my tika baseline this morning and the Matcher
> stopped matching the <meta> tags. Any idea what my be causing this? I've
> tried many variations of the xpath expressions to match the <meta> tags.
> Right now my code in HTMLParser looks like this:
>
> Matcher body = xpath.parse("/HTML/BODY//node()");
> Matcher title = xpath.parse("/HTML/HEAD/TITLE//node()");
> Matcher meta = xpath.parse("/HTML/HEAD/META//node()");
> handler = new TeeContentHandler(
> new MatchingContentHandler(getBodyHandler(xhtml), body),
> new MatchingContentHandler(getTitleHandler(metadata),
> title),
> new MatchingContentHandler(getMetaHandler(metadata), meta));
>
> The <meta> handler isn't being called. If I use /HTML/HEAD//node() the
> handler will get called for the <head> and <title> tags but it will skip
> right past the <meta> tags. I know the tika code is seeing the META tags
> because I see the tags trying to be matched in the startElement method of
> MatchingContentHandler. Any ideas?
>
> --Brian
>

Re: HTML tags

2008-09-24T13:17:25Z

I'll submit the updates when I'm done (along with unit tests). I'm having a
problem though. I sync'ed my tika baseline this morning and the Matcher
stopped matching the <meta> tags. Any idea what my be causing this? I've
tried many variations of the xpath expressions to match the <meta> tags.
Right now my code in HTMLParser looks like this:

Matcher body = xpath.parse("/HTML/BODY//node()");
Matcher title = xpath.parse("/HTML/HEAD/TITLE//node()");
Matcher meta = xpath.parse("/HTML/HEAD/META//node()");
handler = new TeeContentHandler(
new MatchingContentHandler(getBodyHandler(xhtml), body),
new MatchingContentHandler(getTitleHandler(metadata),
title),
new MatchingContentHandler(getMetaHandler(metadata), meta));

The <meta> handler isn't being called. If I use /HTML/HEAD//node() the
handler will get called for the <head> and <title> tags but it will skip
right past the <meta> tags. I know the tika code is seeing the META tags
because I see the tags trying to be matched in the startElement method of
MatchingContentHandler. Any ideas?

--Brian

On Tue, Sep 23, 2008 at 6:04 PM, Thorsten Scherler <thorsten@...>wrote:

> On Sat, 2008-09-20 at 22:41 +0200, Jukka Zitting wrote:
> > Hi,
> >
> > On Fri, Sep 19, 2008 at 7:16 PM, Brian Levay <brian.levay@...>
> wrote:
> > > I need to enhance the functionality of HTMLParser to return the HTML
> <meta>
> > > tags found in the document in the Metadata object. Is overriding
> HTMLParser
> > > (or installing a customer HTMLParser) the best way to do this?
> >
> > We would be happy to receive a patch that adds this feature directly
> > in Tika. :-)
>
> +1
>
> salu2
> >
> > BR,
> >
> > Jukka Zitting
> --
> Thorsten Scherler thorsten.at.apache.org
> Open Source Java consulting, training and solutions
>
>

Re: HTML tags

2008-09-23T15:04:05Z

On Sat, 2008-09-20 at 22:41 +0200, Jukka Zitting wrote:
> Hi,
>
> On Fri, Sep 19, 2008 at 7:16 PM, Brian Levay <brian.levay@...> wrote:
> > I need to enhance the functionality of HTMLParser to return the HTML <meta>
> > tags found in the document in the Metadata object. Is overriding HTMLParser
> > (or installing a customer HTMLParser) the best way to do this?
>
> We would be happy to receive a patch that adds this feature directly
> in Tika. :-)

+1

salu2
>
> BR,
>
> Jukka Zitting
--
Thorsten Scherler thorsten.at.apache.org
Open Source Java consulting, training and solutions

[jira] Resolved: (TIKA-163) GUI does not support drag and drop in Gnome or KDE

2008-09-22T16:11:44Z

[ https://issues.apache.org/jira/browse/TIKA-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting resolved TIKA-163.
--------------------------------

Resolution: Fixed
Fix Version/s: 0.2-incubating
Assignee: Jukka Zitting

Patch applied in revision 698032. Thanks!

> GUI does not support drag and drop in Gnome or KDE
> --------------------------------------------------
>
> Key: TIKA-163
> URL: https://issues.apache.org/jira/browse/TIKA-163
> Project: Tika
> Issue Type: Bug
> Components: gui
> Affects Versions: 0.2-incubating
> Environment: Ubuntu 8.04.1, Gnome 2.22.3
> Reporter: Dave Meikle
> Assignee: Jukka Zitting
> Fix For: 0.2-incubating
>
> Attachments: TIKA-163.diff
>
>
> As Gnome/KDE do not represent transfer file lists as DataFlavor.javaFileListFlavor the GUI does not respond to drag and drop events, rendering it useless in these desktop environments. As they represent transfer file lists as a list of URIs in a String a workaround is to convert this to a list of files.
> A patch will follow to implement this.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (TIKA-140) HTML parser unable to extract text

2008-09-22T16:03:44Z

[ https://issues.apache.org/jira/browse/TIKA-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting resolved TIKA-140.
--------------------------------

Resolution: Fixed

Resolved in a somewhat different manner in revision 698028.

Instead of adding the special "*" wildcard to the XPath matcher, I created a new XHTMLDowngradeHandler decorator class that makes sure that all incoming (X)HTML is uniformly structured.

> HTML parser unable to extract text
> -----------------------------------
>
> Key: TIKA-140
> URL: https://issues.apache.org/jira/browse/TIKA-140
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.2-incubating
> Reporter: julien nioche
> Assignee: Jukka Zitting
> Fix For: 0.2-incubating
>
> Attachments: 1.html, anynamespace.diff
>
>
> At revision 648732
> The file in attachment is not parsed properly by the current HTML parser which returns an empty string when calling ParseUtils.getStringContent(). Saving the same document as .txt from Firefox gives some text.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: New UIMA annotator based on Tika

2008-09-22T14:25:49Z

Hi,

On Mon, Sep 22, 2008 at 10:58 AM, Julien Nioche
<lists.digitalpebble@...> wrote:
> Just to let you know that we've just donated a UIMA component based on Tika
> which is used to convert markup into UIMA annotations, extract the text and
> metadata etc...

Cool, thanks for sharing!

> More details on https://issues.apache.org/jira/browse/UIMA-1095

I gave a quick look at the code and noticed that you apparently need
to sanitize (clean out control characters, normalize spaces) some of
the parsed text output from Word documents. I guess that's something
that we could and should do already in Tika itself.

BR,

Jukka Zitting

[jira] Updated: (TIKA-163) GUI does not support drag and drop in Gnome or KDE

2008-09-22T13:07:44Z

[ https://issues.apache.org/jira/browse/TIKA-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dave Meikle updated TIKA-163:
-----------------------------

Attachment: TIKA-163.diff

Patch to implement workaround.

> GUI does not support drag and drop in Gnome or KDE
> --------------------------------------------------
>
> Key: TIKA-163
> URL: https://issues.apache.org/jira/browse/TIKA-163
> Project: Tika
> Issue Type: Bug
> Components: gui
> Affects Versions: 0.2-incubating
> Environment: Ubuntu 8.04.1, Gnome 2.22.3
> Reporter: Dave Meikle
> Attachments: TIKA-163.diff
>
>
> As Gnome/KDE do not represent transfer file lists as DataFlavor.javaFileListFlavor the GUI does not respond to drag and drop events, rendering it useless in these desktop environments. As they represent transfer file lists as a list of URIs in a String a workaround is to convert this to a list of files.
> A patch will follow to implement this.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-163) GUI does not support drag and drop in Gnome or KDE

2008-09-22T13:03:44Z

GUI does not support drag and drop in Gnome or KDE
--------------------------------------------------

Key: TIKA-163
URL: https://issues.apache.org/jira/browse/TIKA-163
Project: Tika
Issue Type: Bug
Components: gui
Affects Versions: 0.2-incubating
Environment: Ubuntu 8.04.1, Gnome 2.22.3
Reporter: Dave Meikle

As Gnome/KDE do not represent transfer file lists as DataFlavor.javaFileListFlavor the GUI does not respond to drag and drop events, rendering it useless in these desktop environments. As they represent transfer file lists as a list of URIs in a String a workaround is to convert this to a list of files.

A patch will follow to implement this.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.