Spamfilter-Service

[bones0]

Hello all

Even though Akismet was faster than I I still decided to continue with
spfgblib  ;-)  Now I have built a simple page which should be usable as an
API for automated Spam-checking. Here it is:
http://www.chabis.ch/helpers/api/index.php

For trying it out you need a valid userid. Ther is "tester" which can
only check if a text is known SPAM. If you want to get a higher level
which can also train the filter please write your application and tell
me what you want to do (i.E. integrate it into an extension).
Then I may give you a personal ID.

If you want to become part of the project you may also go to
http://www.sourceforge.net/projects/spfgblib

Regards
    Peter


_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Georg Ringer]

Peter Guhl wrote:
> If you want to become part of the project you may also go to
> http://www.sourceforge.net/projects/spfgblib

And how is this related to TYPO3?

georg
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Georg Ringer schrieb:

> Peter Guhl wrote:
>  
>> If you want to become part of the project you may also go to
>> http://www.sourceforge.net/projects/spfgblib
>>    
>
> And how is this related to TYPO3?
>
> georg
>  

OK... probably it's related when I (or somebody else, but I know that
the guy having the idea normally is the one having to do the job)
started to program a typo3-module which is using it. But you'r right -
it's a bit early.

Regards
    Peter

--

mit freundlichen Grüssen

Peter Guhl
Win-Lux GmbH
Obere Schöntalstrasse 22
CH - 8406 Winterthur

Telefon: [phone]
Mobil [phone]
Fax: [phone]
Mail: peter.guhl@...
Skype: p_guhl
Web: http://www.win-lux.ch

******************* Vertraulichkeitserklärung ***********************
Diese Mitteilung resp. deren Beilagen können vertrauliche und/oder
persönliche Informationen enthalten, die einzig für den adressierten
Empfänger bestimmt sind. Jegliche Kenntnisnahme oder Weiterleitung durch
oder an Dritte ist untersagt. Wenn Sie nicht der adressierte Empfänger sind,
nehmen Sie bitte Kontakt mit dem Absender der Nachricht auf und löschen
Sie alle Kopien. Vielen Dank.

_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Hi
Georg Ringer schrieb:
> And how is this related to TYPO3?
>  
Yeah, all I got from 2 mailing lists is a guy complaining. Well, after
all, my spam problem is solved and I know quite many things to spend my
nights with.

Regards
    Peter
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Georg Ringer]

Hi,
> Yeah, all I got from 2 mailing lists is a guy complaining. Well, after
> all, my spam problem is solved and I know quite many things to spend my
> nights with.

I don't complain, I just ask!
This is a TYPO3 mailinglist, so there should be a connection to TYPO3.
If not, this is maybe better in typo3.offtopic!

Probably this is also why you don't get more response here.

Georg
PS: I won't ask my magento questions here because response won't be big ^^
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Dmitry Dulepov [typo3]]

Hi!

Peter Guhl wrote:
> Yeah, all I got from 2 mailing lists is a guy complaining. Well, after
> all, my spam problem is solved and I know quite many things to spend my
> nights with.

I'd love to see an external additional spam filter for comments extension. Can it be done with your library? In general I need to call something that will return certain amount of points. It is approximately like SpamAssassin. Comments extension either publishes comment or sends it to review based on the number of points.

There is a hook for it in comments extension for this already.

--
Dmitry Dulepov
TYPO3 Core team
More about TYPO3: http://typo3bloke.net/
Subscribe: http://typo3bloke.net/rss.xml
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Hi

Georg Ringer schrieb:
> I don't complain, I just ask!
>  
OK. Sorry. Most people ask only if they complain ;-)
> This is a TYPO3 mailinglist, so there should be a connection to TYPO3.
> If not, this is maybe better in typo3.offtopic!
>  
Uhm... didn't know there's such a list. Sorry. At the other hand...
where in the webdesigning-world is spam protection "off topic"?

I definitely have to be picky about the lists I subscribe to - otherwise
my mailbox will soon explode. :-(
> Probably this is also why you don't get more response here.
>  
Maybe...  or nobody has got the time to make it an extension and all
fear that they might need to work if they say something ;-)
> Georg
> PS: I won't ask my magento questions here because response won't be big ^^
>  
Ehr... what's magento? Oh, google knew it ;-) Uhm... if you want to
include it into typo3 I don't know why it was a problem to tell people
around her. But I am probably not the right one to decide what's
appropriate in here and what's not.

Well, my posting wasn't meant to be a question but more a proposal for
something which might be useful for several typo3-extensions.

Unfortunately all I try to do will be rather dirty since my experience
in the typo3-codebase is virtually non-existing...

Regards
     Peter
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Hi

Well, sorry for sounding so grumpy... definitely wasn't meant as a
personal offense to anybody.

Dmitry Dulepov [typo3] schrieb:
> I'd love to see an external additional spam filter for comments
> extension. Can it be done with your library? In general I need to call
> something that will return certain amount of points. It is
> approximately like SpamAssassin. Comments extension either publishes
> comment or sends it to review based on the number of points.
I will have to change the lib to transmit score instead of the boolean.
But that's possible since spamprobe (currently the only software I am
using - even thoug I did, of course, think about including spamassassin)
is sending both.
> There is a hook for it in comments extension for this already.
I'll have to look into it! What number do you need? Spamprobe sends
numbers from 0..1. This can, for example, easily changed into 1..100.

Regards
   Peter

_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Daniel Pšötzinger]

Dmitry Dulepov [typo3] schrieb:

> Hi!
>
> Peter Guhl wrote:
>> Yeah, all I got from 2 mailing lists is a guy complaining. Well, after
>> all, my spam problem is solved and I know quite many things to spend
>> my nights with.
>
> I'd love to see an external additional spam filter for comments
> extension. Can it be done with your library? In general I need to call
> something that will return certain amount of points. It is approximately
> like SpamAssassin. Comments extension either publishes comment or sends
> it to review based on the number of points.

The extension "spamdetection" was planed for this,
(used as a service for other extensions)

maybe its relevant
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Hello

Daniel Pšötzinger schrieb:
> Dmitry Dulepov [typo3] schrieb:
>  
>> something that will return certain amount of points. It is approximately
>> like SpamAssassin. Comments extension either publishes comment or sends
>> it to review based on the number of points.
>>    
> The extension "spamdetection" was planed for this,
> (used as a service for other extensions)
>  
It might be useful since it is one single place for everything and, even
more, it might be an aggregator of more than one spam detection service.
Take akismet, spfgblib and others and calculate a score (and even get a
result if one of the services is currently down).

I changed my API the way it now returns the score. As a number between 0
and 1 (as spamprobe is calculating it out of the box). Writing "SCORE: "
in front of it should make it easy to parse (well, yeah, I know, XML
would be the modern and shiny way... but I get the feeling that " SCORE:
0.5158187" is using less bandwith ;-)

Regards
     Peter
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Michael Stucki-2]

Hi Peter,

> Even though Akismet was faster than I I still decided to continue with
> spfgblib  ;-)  Now I have built a simple page which should be usable as an
> API for automated Spam-checking. Here it is:
> http://www.chabis.ch/helpers/api/index.php
>
> For trying it out you need a valid userid. Ther is "tester" which can
> only check if a text is known SPAM. If you want to get a higher level
> which can also train the filter please write your application and tell
> me what you want to do (i.E. integrate it into an extension).
> Then I may give you a personal ID.
>
> If you want to become part of the project you may also go to
> http://www.sourceforge.net/projects/spfgblib

Sounds interesting, especially since buzz.typo3.org has problems with Spam
and it seems from our side like it's a problem of Akismet...

I'd be happy to see an integration of your library in the ve_guestbook
extension.

- michael
--
Use a newsreader! Check out
http://typo3.org/community/mailing-lists/use-a-news-reader/
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Dmitry Dulepov [typo3]]

Hi!

Peter Guhl wrote:
> Ehr... what's magento? Oh, google knew it ;-) Uhm... if you want to
> include it into typo3 I don't know why it was a problem to tell people
> around her. But I am probably not the right one to decide what's
> appropriate in here and what's not.

I'd love to see magento in TYPO3 :)

--
Dmitry Dulepov
TYPO3 Core team
More about TYPO3: http://typo3bloke.net/
Subscribe: http://typo3bloke.net/rss.xml
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Michael Stucki-2]

Hi Peter,

> Yeah, all I got from 2 mailing lists is a guy complaining. Well, after
> all, my spam problem is solved and I know quite many things to spend my
> nights with.

I'm sure you will get used to that... ;-)

- michael
--
Use a newsreader! Check out
http://typo3.org/community/mailing-lists/use-a-news-reader/
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Dmitry Dulepov [typo3]]

Hi!

Peter Guhl wrote:
> I'll have to look into it! What number do you need? Spamprobe sends
> numbers from 0..1. This can, for example, easily changed into 1..100.

It is from 0 to infinity, integer numbers but completely voluntary in meaning :) To give you an idea: currently comments extension gives one spam point for each three "http://" in text and one spam point per "[url". These two are mostly used by spammers. So, if you have certain spam evaluation criterias, you can construct something of them.

There is also a concept of "cut off" point (obtainable from $pObj->conf['spamProtect.']['spamCutOffPoint']). If you supply a number more than this point, comment will silently dropped.

--
Dmitry Dulepov
TYPO3 Core team
More about TYPO3: http://typo3bloke.net/
Subscribe: http://typo3bloke.net/rss.xml
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Dmitry Dulepov [typo3]]

Hi!

Daniel Pšötzinger wrote:
> The extension "spamdetection" was planed for this,
> (used as a service for other extensions)
>
> maybe its relevant

Good to know, thanks!

--
Dmitry Dulepov
TYPO3 Core team
More about TYPO3: http://typo3bloke.net/
Subscribe: http://typo3bloke.net/rss.xml
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Hi

Dmitry Dulepov [typo3] schrieb:
> It is from 0 to infinity, integer numbers but completely voluntary in meaning :) To give you an idea: currently comments extension gives one spam point for each three "http://" in text and one spam point per "[url". These two are mostly used by spammers. So, if you have certain spam evaluation criterias, you can construct something of them.
>  
Well, that's probably what the bayes filter internally does.

Now... how should I achieve that... I get points from 0.00000000 to
0.99999999. Harmless text range around 0.5 or a bit above. If I
"normalize" that to 1..10 I get points a bit higher than spamassassin
(From what I know there score 3 is normally SPAM, above 10 it's always).

No matter what I do; the behaviour will be different from Spamassassin.
I really think about the extension "spamdetection" which might be good
to level several sources the way they give a reliable scoring system.
Thats the way spamassassin works. The Bayes-Filter there is giving out a
percentage and SA seems to give them points based on percentage-ranges
(i.e. 0 to 10% = -1 Point, 10 to 30% 0.1 Point, 30 to 50 % 0.5 Points...).

Of course any extension might access spfgblib directly as long as it
takes in account the way it behaves.

I installed spamdetection. From the first look it apparently uses rules
and blacklists. That's a good idea and can be combined with external
sources. Does it already do scoring? I think I will do my first steps
touse the ASP spfgblib somewhere with that extension. The resulting
thing will probably be a PHP-snippet which should be usable elsewhere too.
> There is also a concept of "cut off" point (obtainable from $pObj->conf['spamProtect.']['spamCutOffPoint']). If you supply a number more than this point, comment will silently dropped.
>  
If you work with score you need that, of course.

Regards
      Peter

_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Hi

Michael Stucki schrieb:
>> Yeah, all I got from 2 mailing lists is a guy complaining. Well, after
>> all, my spam problem is solved and I know quite many things to spend my
>> nights with.
>>    
>
> I'm sure you will get used to that... ;-)
>  
It's about normal, I know. But sometimes I can't help to state it again
;-) And I am really happy that I got it wrong this time!

Regards
   Peter

_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Daniel Pšötzinger schrieb:
> The extension "spamdetection" was planed for this,
> (used as a service for other extensions)
>  
I sort of managed to install the application (well, the files are
there...). Now, the whole thing looks very straight forward. All I would
have to do seems to be to add a few lines into getSpamPercentForText()
which do a http-Request to my API-Form and get the number out of the
result (maybe with a few multiplications - we'll see). It might only get
active when you call getSpamPercentForText() with a pseudo-ruleset like
"onlinespfgblib" (that way it can be ommitted if you don't want to waste
bandwith).

What do you think? Is that the right track?

Regards
      Peter
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[bones0]

Hi Daniel

Daniel Pšötzinger schrieb:
> The extension "spamdetection" was planed for this,
Well, I gave it a try and inserted the following lines below line 219 of
class.tx_spamdetection_service.php:

   
        // First test: Send a http-Request to the spfgblib-API
       
$spfgblib=http_post_fields('http://www.chabis.ch/helpers/api/index.php',array('sent'=>'spamform','userid'=>'tester','test'=>'TEST','text'=>$text));
        $spfgbscore=substr($spfgblib,strstr($spfgblib,'SCORE: ')+7,9);
        $percents[]=$spfgbscore * 100;

Well.. now I have to find out how I can test it ;-)

Regards
    Peter
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev