Applying a costmatrix in the Experimenter

Hi,

 

My problem is the following. I would like to use the WEKA Experimenter to analyse my dataset, which is highly unbalanced.

In order to overcome this imbalance I want to apply a costmatrix, where costs of a FP prediction are higher then costs of a FN prediction.

 

However, when selecting in the Experimenter the following:

Result Generator = CrossValidation Result Producer à Split Evaluator = CostSensitiveClassifierSplitEvaluator à Classifier = CostSensitiveClassifier

You get to the final window where you can program the base classifier (J48 in my case) and you can define a costmatrix. By default is mentions a 1x1 cost matrix.

As my output has two classes I change the costmatrix in a 2x2 contingency table and change the appropriate cost values. The costmatrix source then automatically changes into Use Explicit cost matrix. After setting this experiment (with this costmatrix) I run the experiment with a dataset called: fold0TR.arff

 

The experiment however is stopped due to an error being: On-demand cost file doesn't exist: C:\Program Files\Weka-3-4\fold0TR.cost

Does anybody know why I get this error warning, and how I can prevent it from occurring?

Thanks in advance!

 

Kind regards,

 

 

Claudia

 

Let op: mijn email adres is vanaf nu C.Kamphuis@...

 

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

No right, this setup is for the property iterator. Normally, one
doesn't have to change that at all. Instead of using the advanced
setup, use the simple setup.

Your setup is wrong. Do the following instead:
- use the simple setup, not the advanced one
- make sure cross-validation is selected as "Experiment type"
- add your dataset "fold0TR.arff"
- add the CostSensitiveClassifier to the algorithms list
  (just setup the cost-matrix the way you did it before)
- adjust CV parameters if necessary (folds, runs)
- run and evaluate the experiment

HTH

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/ +64 (7) 838-4466 Ext. 5174

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Thanks Peter for your comments about applying a costmatrix in the Experimenter.
It sure works, but I do have some additional problems:

1) Is it possible in the Experimenter to build cost-sensitive trees using more training sets (self-defined) and test these trees on separate test files (also self-defined) so without letting the Experimenter conducting 10f cross-validation. Maybe I have to select another "Experiment type" as the minimum number of folds in the Cross-Validation is 2.
2) when I use the KnowlegdeFlow in WEKA I can produce a 'sinkfile' where I can append prediction for each class as well. Is it possible to do such a thing in the Experimenter as well?
3) Do you perhaps know as well why it takes so long to import the .csv file, created in the KnowlegdFlow as sinkfile, into SAS?

Hopefully you (or of course anybody else) can help me with these queries as well. Thanks in advance.

Groetjes,


Claudia

-----Oorspronkelijk bericht-----
Van: wekalist-bounces@... [mailto:wekalist-bounces@...] Namens Peter Reutemann
Verzonden: vrijdag 21 maart 2008 0:12
Aan: Weka machine learning workbench list.
Onderwerp: Re: [Wekalist] Applying a costmatrix in the Experimenter

No right, this setup is for the property iterator. Normally, one
doesn't have to change that at all. Instead of using the advanced
setup, use the simple setup.

Your setup is wrong. Do the following instead:
- use the simple setup, not the advanced one
- make sure cross-validation is selected as "Experiment type"
- add your dataset "fold0TR.arff"
- add the CostSensitiveClassifier to the algorithms list
  (just setup the cost-matrix the way you did it before)
- adjust CV parameters if necessary (folds, runs)
- run and evaluate the experiment

HTH

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/ +64 (7) 838-4466 Ext. 5174

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

> Thanks Peter for your comments about applying a costmatrix in the Experimenter.
>  It sure works, but I do have some additional problems:
>
>  1) Is it possible in the Experimenter to build cost-sensitive trees using more training sets (self-defined) and test these trees on separate test files (also self-defined) so without letting the Experimenter conducting 10f cross-validation. Maybe I have to select another "Experiment type" as the minimum number of folds in the Cross-Validation is 2.

You can't specify any other test sets.

>  2) when I use the KnowlegdeFlow in WEKA I can produce a 'sinkfile' where I can append prediction for each class as well. Is it possible to do such a thing in the Experimenter as well?

No, the Experimenter only collects the statistics, not the predictions.

>  3) Do you perhaps know as well why it takes so long to import the .csv file, created in the KnowlegdFlow as sinkfile, into SAS?

I assume SAS is a third-party application and it sounds like the
problem lies at the SAS end.

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/ +64 (7) 838-4466 Ext. 5174

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Do you have null values in your dataset? Weka dumps null values as ?'s when saving csv's. If you import that in SAS, numeric attributes will be converted to text -- resulting in a useless dataset that is also bigger in size and takes longer to import.  

-----Original Message-----
From: "Peter Reutemann" <fracpete@...>
To: "Weka machine learning workbench list." <wekalist@...>
Sent: 3/26/2008 2:53 PM
Subject: Re: [Wekalist] Applying a costmatrix in the Experimenter

> Thanks Peter for your comments about applying a costmatrix in the Experimenter.
>  It sure works, but I do have some additional problems:
>
>  1) Is it possible in the Experimenter to build cost-sensitive trees using more training sets (self-defined) and test these trees on separate test files (also self-defined) so without letting the Experimenter conducting 10f cross-validation. Maybe I have to select another "Experiment type" as the minimum number of folds in the Cross-Validation is 2.

You can't specify any other test sets.

>  2) when I use the KnowlegdeFlow in WEKA I can produce a 'sinkfile' where I can append prediction for each class as well. Is it possible to do such a thing in the Experimenter as well?

No, the Experimenter only collects the statistics, not the predictions.

>  3) Do you perhaps know as well why it takes so long to import the .csv file, created in the KnowlegdFlow as sinkfile, into SAS?

I assume SAS is a third-party application and it sounds like the
problem lies at the SAS end.

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/ +64 (7) 838-4466 Ext. 5174

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist


_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Thank you Peter and Fernando for both your answers!
It helped me a lot (at least to decide not to spend to much time in the
experimenter).

Yes, I do have a lot of missing values in the dataset (and thus in the
.csv files).
I'll have a look what I can do with it!

Again, thanks!

Groetjes,
Claudia


-----Oorspronkelijk bericht-----
Van: wekalist-bounces@...
[mailto:wekalist-bounces@...] Namens Fernando Cela
Diaz
Verzonden: donderdag 27 maart 2008 4:59
Aan: Weka machine learning workbench list.
Onderwerp: RE: [Wekalist] Applying a costmatrix in the Experimenter

Do you have null values in your dataset? Weka dumps null values as ?'s
when saving csv's. If you import that in SAS, numeric attributes will be
converted to text -- resulting in a useless dataset that is also bigger
in size and takes longer to import.  

-----Original Message-----
From: "Peter Reutemann" <fracpete@...>
To: "Weka machine learning workbench list."
<wekalist@...>
Sent: 3/26/2008 2:53 PM
Subject: Re: [Wekalist] Applying a costmatrix in the Experimenter

> Thanks Peter for your comments about applying a costmatrix in the
Experimenter.
>  It sure works, but I do have some additional problems:
>
>  1) Is it possible in the Experimenter to build cost-sensitive trees
using more training sets (self-defined) and test these trees on separate
test files (also self-defined) so without letting the Experimenter
conducting 10f cross-validation. Maybe I have to select another
"Experiment type" as the minimum number of folds in the Cross-Validation
is 2.

You can't specify any other test sets.

>  2) when I use the KnowlegdeFlow in WEKA I can produce a 'sinkfile'
where I can append prediction for each class as well. Is it possible to
do such a thing in the Experimenter as well?

No, the Experimenter only collects the statistics, not the predictions.

>  3) Do you perhaps know as well why it takes so long to import the
.csv file, created in the KnowlegdFlow as sinkfile, into SAS?

I assume SAS is a third-party application and it sounds like the
problem lies at the SAS end.

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/ +64 (7) 838-4466 Ext. 5174

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist


_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist


_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

I'm afraid that there isn't any facility for computing partial AUC in  
Weka. The AUC in Weka is computed using the non-parametric Wilcoxon-
Mann-Whitney test.

Cheers,
Mark.

On 27/06/2008, at 11:59 PM, Kamphuis, C. (Claudia) wrote:
--
Mark Hall
Senior Developer/Consultant, Pentaho Open Source Business Intelligence
Citadel International, Suite 340, 5950 Hazeltine National Dr.,
Orlando, FL 32822, USA
+64 7 847-3537 office, +64 21 399-132 mobile, +1 815 550-8637 fax,
Skype: mark.andrew.hall, Yahoo: mark_andrew_hall
Download the latest release today <http://www.sourceforge.net/ 
projects/pentaho>




_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Hi Mark,

Thank you anyways for your reply. This makes me stop trying to look for
a way in WEKA. Thanks also for your comment on how WEKA computes AUC.

Thanks.
Claudia

-----Oorspronkelijk bericht-----
Van: wekalist-bounces@...
[mailto:wekalist-bounces@...] Namens Mark Hall
Verzonden: dinsdag 1 juli 2008 0:59
Aan: Weka machine learning workbench list.
Onderwerp: Re: [Wekalist] computing a partical area under the curve

I'm afraid that there isn't any facility for computing partial AUC in  
Weka. The AUC in Weka is computed using the non-parametric Wilcoxon-
Mann-Whitney test.

Cheers,
Mark.

On 27/06/2008, at 11:59 PM, Kamphuis, C. (Claudia) wrote:
--
Mark Hall
Senior Developer/Consultant, Pentaho Open Source Business Intelligence
Citadel International, Suite 340, 5950 Hazeltine National Dr.,
Orlando, FL 32822, USA
+64 7 847-3537 office, +64 21 399-132 mobile, +1 815 550-8637 fax,
Skype: mark.andrew.hall, Yahoo: mark_andrew_hall
Download the latest release today <http://www.sourceforge.net/ 
projects/pentaho>





_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Hi

The Wilcoxon Mann-Whitnay calculation is equivalent to the trapezoid rule
for the area (depens on step size). This allows to use partial areas and
normalise the area for these ( divide by the maximum possible partial area).
In older versions the Weka AUC calculation used the trapezoid rule and
summed the curve segment areas.

Aletrnatively port the weka threshold curve to an appropiate ROC regression
algorithm. It will be hard to determine the accuracy of the partial AUC
especially for FP = 0.03. How many cases does that contain?


Hans van Rijnbrek




At 10:59 01/07/2008 +1200, Mark Hall wrote: Hans v Rijnberk
JRijnberk@...




_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Hi Hans,

The number of cases where FP<=0.03 includes 147 cases (89 negatives, 58
positives).
I am very sorry to say that I find it hard to understand what you're
saying.

What I have tried so far is the following (which probably seems very
inefficient):
When using the Explorer in WEKA, and after implementing a J48 on a
dataset (highly imbalanced: 3000negs and 100pos) with 10fcv I look at
the ROC-curve (which also mentions the total AUC). I save the output
from this curve, which is an .arff file mentioning the
False_Positive_Rate and True_Positive_Rate for each threshold.

In addition, I also make a file from the Explore results including a
probability of having the disease for each instance.

The next step is to look in the First file which threshold is used at a
False_positive_Rate van 0.03. It is this threshold that I use in the
second file to select only those instances that have a value equal or
larger than the threshold value. This selection of instances are the
ones I run through a sas macro (%ROC, a 'standard' macro within SAS to
compute the Total AUC), and the result is, I think, the partial AUC.
However, I don't know whether this is a statistically correct approach
(which I doubt: it seems almost too simple).

Hopefully, you understand what I am trying to do (assuming you also have
knowledge about WEKA and its outputs), and of course hopefully you can
help me further!

Thanks in advance,

Claudia Kamphuis



-----Oorspronkelijk bericht-----
Van: wekalist-bounces@...
[mailto:wekalist-bounces@...] Namens
JRijnberk@...
Verzonden: dinsdag 1 juli 2008 10:12
Aan: Weka machine learning workbench list.; Weka machine learning
workbench list.
Onderwerp: Re: [Wekalist] computing a partical area under the curve

Hi

The Wilcoxon Mann-Whitnay calculation is equivalent to the trapezoid
rule
for the area (depens on step size). This allows to use partial areas and
normalise the area for these ( divide by the maximum possible partial
area).
In older versions the Weka AUC calculation used the trapezoid rule and
summed the curve segment areas.

Aletrnatively port the weka threshold curve to an appropiate ROC
regression
algorithm. It will be hard to determine the accuracy of the partial AUC
especially for FP = 0.03. How many cases does that contain?


Hans van Rijnbrek




At 10:59 01/07/2008 +1200, Mark Hall wrote: Hans v Rijnberk
JRijnberk@...





_______________________________________________
Wekalist mailing list
Wekalist@...
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist