MarkMail: Powered by Apache James

[Jason Hunter-3]

Some of you may have seen my announcement this morning on community@
about MarkMail, a new email archiving service.

http://markmail.org/message/q2dtugicwt6kzfdj

Apache James lives at the heart of the email ingestion pipeline, and I
wanted to write in here to say thanks to the people who built such a
useful tool.  Having a Java-based extensible mail processor really
simplified my life.  If you see me at ApacheCon, let me know: I've got a
T-shirt for you.

Funny story: It was when hitting some odd James behavior that MarkMail
had its first "self hosting" moment.  I noticed I could move files
manually into the spool directory but they wouldn't be recognized until
after a server restart.  I searched on Google: "james server restart",
"apache james server restart", and things like this.  No luck.  James is
just too common a word.  Then I remembered we'd loaded all the Apache
emails onto our test system!  Duh.  So I tried "list:james restart respool".

http://markmail.org/search/?q=list%3Ajames+restart+respool

Bingo!

I do have a lingering question whose answer I couldn't find in the archive:

* Is there any way to easily employ multiple match conditions
   (subject is X, sender contains Y) without setting up extra processors?

-jh-

[Stefano Bagnara-2]

Jason Hunter ha scritto:
> Some of you may have seen my announcement this morning on community@
> about MarkMail, a new email archiving service.
> [...]
> Apache James lives at the heart of the email ingestion pipeline, and I
> wanted to write in here to say thanks to the people who built such a
> useful tool.  Having a Java-based extensible mail processor really
> simplified my life.  If you see me at ApacheCon, let me know: I've got a
> T-shirt for you.
> [...]

Cool! Thank you for sharing your experience!

> I do have a lingering question whose answer I couldn't find in the archive:
>
> * Is there any way to easily employ multiple match conditions
>   (subject is X, sender contains Y) without setting up extra processors?

No, unfortunately there's no way. Programmatically it would be easy to
support something similar, but it's hard to find out an easy xml
configuration to define complex expressions, so we never added this feature.

Stefano

[Danny Angus]

Jason,

Great stuff! I love it when people use James :-) Most people seem to
quietly get on with it, thanks for sharing with us. I already loved
MarkMail, this makes me love it even more!

Unfortunately I won't be at ACUS, and I've had to pull my James talk :-(

As for complex conditions you might try JSeive, or alternatively write
a rules based matcher yourself.

We did, at one time, have *some* ideas for constructing nesting and
conditionality in matcher configs, but it hasn't got off the ground.
@Work we recently developed an xml schema for constructing complex
conditions, perhaps I should resurrect the idea.

d.

[JAMES Nightly Build System]

> Unfortunately I won't be at ACUS, and I've had to pull my James talk :-(

Aren't I doing it for you?

        --- Noel

[JAMES Nightly Build System]

Noel J. Bergman wrote:

> > Unfortunately I won't be at ACUS, and I've had to pull my James talk :-(
> Aren't I doing it for you?

Seems that I am: http://us.apachecon.com/us2007/program/talk/1861.  I still need your slides.

Jason, do you want to guest-star and add some information (ad hoc if you want) about how JAMES is core to MarkMail?

        --- Noel

[Danny Angus]

On 08/11/2007, Noel J. Bergman <noel@...> wrote:

> > Aren't I doing it for you?

Oh!, I thought that they were pretty down on that.
>
> Seems that I am: http://us.apachecon.com/us2007/program/talk/1861.  I still need your slides.

You'll need my notes too?

>
> Jason, do you want to guest-star and add some information (ad hoc if you want) about how JAMES is core to MarkMail?
>
>         --- Noel
>
>
>
>

[JAMES Nightly Build System]

Danny Angus wrote:

> Noel J. Bergman <noel@...> wrote:
> > Aren't I doing it for you?
> Oh!, I thought that they were pretty down on that.

Somewhat to my surprise after the initial comments, it was decided to run it, anyway.

> > Seems that I am: http://us.apachecon.com/us2007/program/talk/1861.
> > I still need your slides.
> You'll need my notes too?

Whatever you've got.

        --- Noel

[Jason Hunter-3]

Noel J. Bergman wrote:
> Noel J. Bergman wrote:
>
>>> Unfortunately I won't be at ACUS, and I've had to pull my James talk :-(
>> Aren't I doing it for you?
>
> Seems that I am: http://us.apachecon.com/us2007/program/talk/1861.  I still need your slides.
>
> Jason, do you want to guest-star and add some information (ad hoc if you want) about how JAMES is core to MarkMail?

Oh sure, I'd enjoy that.  I was already planning to attend that talk.

-jh-

[Jason Hunter-3]

Danny Angus wrote:
> As for complex conditions you might try JSeive, or alternatively write
> a rules based matcher yourself.

Thanks for letting me know I wasn't just overlooking something.  I think
writing my own matcher would suffice for my needs.

FYI, that reminds me, I wrote my own MboxMailRepository.  I didn't
contribute it back because mine only needed to be write-only (I write
every incoming email to Mark Logic and an mbox file).  In my mbox
version I fixed a few issues with the existing one:

  * Reports exceptions if the mbox dir isn't there.
  * Closes files in finally block.
  * Nicely exceptions out if no disk space.
  * Ensures "envsender" in From line is one word.

I'd be happy to send in the rewritten version if someone wants to put
these fixes into the trunk.

-jh-

[Danny Angus]

On 08/11/2007, Jason Hunter <jhunter@...> wrote:

> I'd be happy to send in the rewritten version if someone wants to put
> these fixes into the trunk.

Ill do that, all contributions are more than welcome :-)
If you could create an issue in JIRA, attach the patch/file whatever
and assign it to me that would help us not to forget it!

d.

[Norman Maurer-3]

It whould be really cool if you could create an issue on jira and grant
rights, so we can include it to our trunk.

thx

BTW if you are intressted in mbox maybe mstor.sf.net is a good choice
for you too ?

bye
Norman

Am Donnerstag, den 08.11.2007, 10:56 -0800 schrieb Jason Hunter:

> Danny Angus wrote:
> > As for complex conditions you might try JSeive, or alternatively write
> > a rules based matcher yourself.
>
> Thanks for letting me know I wasn't just overlooking something.  I think
> writing my own matcher would suffice for my needs.
>
> FYI, that reminds me, I wrote my own MboxMailRepository.  I didn't
> contribute it back because mine only needed to be write-only (I write
> every incoming email to Mark Logic and an mbox file).  In my mbox
> version I fixed a few issues with the existing one:
>
>   * Reports exceptions if the mbox dir isn't there.
>   * Closes files in finally block.
>   * Nicely exceptions out if no disk space.
>   * Ensures "envsender" in From line is one word.
>
> I'd be happy to send in the rewritten version if someone wants to put
> these fixes into the trunk.
>
> -jh-

[Norman Maurer-3]

Am Donnerstag, den 08.11.2007, 19:12 +0000 schrieb Danny Angus:

> On 08/11/2007, Jason Hunter <jhunter@...> wrote:
>
> > I'd be happy to send in the rewritten version if someone wants to put
> > these fixes into the trunk.
>
> Ill do that, all contributions are more than welcome :-)
> If you could create an issue in JIRA, attach the patch/file whatever
> and assign it to me that would help us not to forget it!
>
> d.

+1
Norman

[Norman Maurer-3]

Am Donnerstag, den 08.11.2007, 10:46 -0800 schrieb Jason Hunter:

> Noel J. Bergman wrote:
> > Noel J. Bergman wrote:
> >
> >>> Unfortunately I won't be at ACUS, and I've had to pull my James talk :-(
> >> Aren't I doing it for you?
> >
> > Seems that I am: http://us.apachecon.com/us2007/program/talk/1861.  I still need your slides.
> >
> > Jason, do you want to guest-star and add some information (ad hoc if you want) about how JAMES is core to MarkMail?
>
> Oh sure, I'd enjoy that.  I was already planning to attend that talk.
>

I really whould like to be there :-/ Hopefully nexttime on ApacheCon EU
08

bye
Norman

[Jason Hunter-3]

Norman Maurer wrote:
> It whould be really cool if you could create an issue on jira and grant
> rights, so we can include it to our trunk.

Will do.

> BTW if you are intressted in mbox maybe mstor.sf.net is a good choice
> for you too ?

Funny you should mention mstor.  We use it already.  When we bulk load
from an existing mbox archive on the filesystem, we use mstor to parse
the files.  Because both it and James expose the JavaMail API, we can
use the same back-end email-to-xml conversion code.

For testing purposes, I wrote an mstor-driven client that sends each
email in the mbox file through the James listener via SMTP.  I pushed
4,000,000 emails through James one day.  :)

-jh-

[Danny Angus]

On 08/11/2007, Jason Hunter <jhunter@...> wrote:

> 4,000,000 emails through James one day.  :)

That never gets old!

d.

[robert burrell donkin-2]

On Nov 8, 2007 9:02 AM, Stefano Bagnara <apache@...> wrote:

> Jason Hunter ha scritto:
> > Some of you may have seen my announcement this morning on community@
> > about MarkMail, a new email archiving service.
> > [...]
> > Apache James lives at the heart of the email ingestion pipeline, and I
> > wanted to write in here to say thanks to the people who built such a
> > useful tool.  Having a Java-based extensible mail processor really
> > simplified my life.  If you see me at ApacheCon, let me know: I've got a
> > T-shirt for you.
> > [...]
>
> Cool! Thank you for sharing your experience!

+1

(for those who don't know, jason is one of the greats from the last
days of java@apache and the earliest days of jakarta so it's
especially cool to find out what he's been working on)

been meaning to blog about markmail since this concept is one of my
long term email itches - great to see a working implementation :-)

any chance of adding searchable archives for some of the email
specification lists? (apache is a rank 9 site but most of the email
specification archives seem to be buried on low ranked sites and are
effectively invisible to conventional search engines)

> > I do have a lingering question whose answer I couldn't find in the archive:
> >
> > * Is there any way to easily employ multiple match conditions
> >   (subject is X, sender contains Y) without setting up extra processors?
>
> No, unfortunately there's no way. Programmatically it would be easy to
> support something similar, but it's hard to find out an easy xml
> configuration to define complex expressions, so we never added this feature.

a rules engine supporting xml would probably do the job. drools seems
particularly popular right now but most JSR-94 implementation would
probably do the job. i suspect that the work invovled wouldn't be
great. anyone fancy taking this one on?

- robert

[Danny Angus]

On 09/11/2007, Robert Burrell Donkin <robertburrelldonkin@...> wrote:

> drools seems particularly popular right now but most JSR-94 implementation would
> probably do the job. i suspect that the work invovled wouldn't be
> great. anyone fancy taking this one on?

Drools was exactly my first thought, its a wonder that we hadn't
thought of it before.
I'll raise an issue in JIRA so we don't forget. Hopefully I'll be able
to do the job too.

d.

[Jason Hunter-3]

Robert Burrell Donkin wrote:

> any chance of adding searchable archives for some of the email
> specification lists? (apache is a rank 9 site but most of the email
> specification archives seem to be buried on low ranked sites and are
> effectively invisible to conventional search engines)

Like the piano player at the bar, we take requests.  :)

Which exact lists are you referring to?  The more details you can give,
the easier it is to include them.

-jh-

[JAMES Nightly Build System]

> > drools seems particularly popular right now but most JSR-94 implementation would
> > probably do the job.

And don't forget that we already have JSieve.

        --- Noel

[robert burrell donkin-2]

On Nov 9, 2007 11:10 AM, Jason Hunter <jhunter@...> wrote:

> Robert Burrell Donkin wrote:
>
> > any chance of adding searchable archives for some of the email
> > specification lists? (apache is a rank 9 site but most of the email
> > specification archives seem to be buried on low ranked sites and are
> > effectively invisible to conventional search engines)
>
> Like the piano player at the bar, we take requests.  :)
>
> Which exact lists are you referring to?  The more details you can give,
> the easier it is to include them.

http://www3.ietf.org/proceedings/94mar/charters/imap-charter.html

    *  General Discussion <imap@...>
    * To Subscribe <imap-request@...>
    * Archive <ftp.cac.washington.edu:~/imap/imap_archive>

http://www.ietf.org/html.charters/sieve-charter.html

General Discussion: ietf-mta-filters@...
To Subscribe: ietf-mta-filters-request@...
In Body: body=subscribe
Archive: http://www.imc.org/ietf-mta-filters/mail-archive/

http://www.imc.org/ietf-imapext/

would be very useful

but generally, w3c (http://www.w3.org/Mail/) and IEFT  lists
(http://www.ietf.org/WG-WEB-Mail.html) would be great

- robert