Nabble - DDC-Concordance

Re: IndicesToShow not shown in query '#within file'

2008-06-11T03:26:16Z

On Tuesday 10 June 2008 14.32:09 Алексей Сокирко wrote:
> > 1. IndicesToShow not shown in query '#within file' (tobias roth)
>
> That's true it should be fixed in next versions.
>

The same is true for the context operator, I assume?
(with '... #cntxt 1', I get something like:
'Context sentence before. Real#ADJA sentence#NN .#$. Context sentence
after')

Cheers,
tobi

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: Re: Problems installing rpm of version 1.80 on Fedora

2008-06-10T07:05:07Z

Hi Tobi,

thanks for the error report. It seems there is also a rights problem in the package - if i use the root account, indexing bigrams works fine, otherwise not.

I'll fix that as soon as possible.

best,

Kai

On Jun 10, 2008, tobias.roth@... wrote:

On Tuesday 10 June 2008 14.44:50 Kai Zimmer wrote:
> does renaming opt/ddc/Dicts/Obor/DomItems.txt to domitems.txt fix the
> problem? If so, i would release a new rpm...

Not exactly...
After renaming, it asks for 'items.txt'. And as I couldn't find anything
like 'items.txt' in .../Obor, but there is one in .../GerObor, I removed
the whole Obor directory and replaced it by GerObor.

Like this, indexing and start-up both seem to work fine, but maybe this is
not totally right, either: when I add the new Bigrams record, I get the
following error on indexing:

[...]
To file test_utf8._perdivToken 2 vectors were written
Cannot create temp file
cannot index the project!

Cheers,
tobi
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: Problems installing rpm of version 1.80 on Fedora

2008-06-10T06:31:50Z

On Tuesday 10 June 2008 14.44:50 Kai Zimmer wrote:
> does renaming opt/ddc/Dicts/Obor/DomItems.txt to domitems.txt fix the
> problem? If so, i would release a new rpm...

Not exactly...
After renaming, it asks for 'items.txt'. And as I couldn't find anything
like 'items.txt' in .../Obor, but there is one in .../GerObor, I removed
the whole Obor directory and replaced it by GerObor.

Like this, indexing and start-up both seem to work fine, but maybe this is
not totally right, either: when I add the new Bigrams record, I get the
following error on indexing:

[...]
To file test_utf8._perdivToken 2 vectors were written
Cannot create temp file
cannot index the project!

Cheers,
tobi
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: Problems installing rpm of version 1.80 on Fedora

2008-06-10T05:44:50Z

Hi Tobi,

does renaming opt/ddc/Dicts/Obor/DomItems.txt to domitems.txt fix the problem?

If so, i would release a new rpm...

best,

Kai

On Jun 10, 2008, tobias.roth@... wrote:

Hi Kai,

I've installed the ddc-1.80 rpm now on a freshly installed Fedora 9, but
I'm still getting errors on start-up/indexing.

ConcordIndex e.g. fails during indexing saying:

MaxTokenCountInOnePeriod = 5000000
/opt/ddc/Dicts/Obor: cannot find domitems.txt
Graphan::LoadDicts failed! Error: Cannot load oborots
cannot index the project!

In /opt/ddc/Dicts/Obor, there is a file 'DomItems.txt', a
file 'domitems.txt' exists in /opt/ddc/Dicts/GerObor, however.

Cheers,
tobi
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: ddc-users Digest, Vol 7, Issue 1

2008-06-10T05:32:09Z

ddc-users-request@... пишет:

That's true it should be fixed in next versions.

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Problems installing rpm of version 1.80 on Fedora

2008-06-10T05:20:42Z

Hi Kai,

I've installed the ddc-1.80 rpm now on a freshly installed Fedora 9, but
I'm still getting errors on start-up/indexing.

ConcordIndex e.g. fails during indexing saying:

MaxTokenCountInOnePeriod = 5000000
/opt/ddc/Dicts/Obor: cannot find domitems.txt
Graphan::LoadDicts failed! Error: Cannot load oborots
cannot index the project!

In /opt/ddc/Dicts/Obor, there is a file 'DomItems.txt', a
file 'domitems.txt' exists in /opt/ddc/Dicts/GerObor, however.

Cheers,
tobi
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: IndicesToShow not shown in query '#within file'

2008-06-10T03:28:17Z

Hi Kai,

On Tuesday 10 June 2008 11.51:15 Kai Zimmer wrote:
> can you please try the same with ddc_xml?

Same result, unfortunately. That's probably not a good sign, hmm?

> By the way: you should definitely swith to the new ddc-1.80 (available
> on Sourceforge). It contains a fixed bug about the number of hits.

But still on version 1.74 - the rpm for the new version gives me errors
during indexing/start-up, but I have only tried upgrading from an older
version on Fedora 7. I'll first try a clean install on Fedora 9 (and --
hopefully not -- ask again if I still get the same errors).

Cheers,
tobi
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: IndicesToShow not shown in query '#within file'

2008-06-10T02:51:15Z

Hi Tobi,

can you please try the same with ddc_xml?
By the way: you should definitely swith to the new ddc-1.80 (available
on Sourceforge). It contains a fixed bug about the number of hits.

thanks,
Kai

tobias roth schrieb:

> Hi there,
>
> A query with the modifier '#within file' doesn't give me
> the 'IndicesToShow' specified in the opt file.
>
> So:
> ../ddc-client.py --query="immer"
>
> yields:
> [...]
> Der#ART#5070 Marggraf#NN#5071 trägt#VVFIN#5072 zu#APPR#5073 dem#ART#5074
> [...]
>
> whereas:
> ../ddc-client.py --query="immer #within file"
>
> will output:
> [...]
> Der Marggraf trägt zu dem
> [...]

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

IndicesToShow not shown in query '#within file'

2008-06-10T02:36:06Z

Hi there,

A query with the modifier '#within file' doesn't give me
the 'IndicesToShow' specified in the opt file.

So:
./ddc-client.py --query="immer"

yields:
[...]
Der#ART#5070 Marggraf#NN#5071 trägt#VVFIN#5072 zu#APPR#5073 dem#ART#5074
[...]

whereas:
./ddc-client.py --query="immer #within file"

will output:
[...]
Der Marggraf trägt zu dem
[...]

Is there anything I can do about this?

Cheers
tobi

(I'm using ddc 1.74-1, the python client in 'text' mode, and right now the
following opt file:

Generic
IndexType Free_Index
Indices [Token w normal]; [Pos p normal storage]; [Lemma l normal storage];
[Thes t normal]; [Line i normal storage]; [Tags x normal]
HitBorders [s:sentence:default]
IndicesToShow 1 2 5
OutputBibliographyOfHits
ResumeOnIndexErrors
LeftKwicContextSize 20
RightKwicContextSize 20
NumberOfKwicLinesInSnippets 1
TfIdfRank 2.0
NearRank 1.0
PositionRank 1.0
Bibl string 1
textClass /TEI.2/teiHeader/profileDesc/textClass/keywords/term[@type="short_category"]
Bibl string 1
textClassLong /TEI.2/teiHeader/profileDesc/textClass/keywords/term[@type="category"]
Bibl string 1
title /TEI.2/teiHeader/fileDesc/sourceDesc/biblFull/titleStmt/title[@type="abbreviated"]
Bibl integer 0
date /TEI.2/teiHeader/fileDesc/sourceDesc/biblFull/publicationStmt/date[@id="publication_year"]
Bibl string 1
availability /TEI.2/teiHeader/fileDesc/publicationStmt/availability/@status
Bibl string 1 bibliography /TEI.2/teiHeader/fileDesc/sourceDesc/bibl
Bibl string 1
site /TEI.2/teiHeader/fileDesc/publicationStmt/pubPlace[@id="conversion_place"]
Bibl integer 1 page /TEI.2/text/body/pb/@n
Bibl string 1
idno /TEI.2/teiHeader/fileDesc/sourceDesc/biblFull/publicationStmt/idno
Bibl string 1 author /TEI.2/teiHeader/fileDesc/titleStmt/author
Bibl string 1
pubPlace /TEI.2/teiHeader/fileDesc/sourceDesc/biblFull/publicationStmt/pubPlace[@id="publication_place"]
TextArea body /TEI.2/text/body
)

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: ddc-users Digest, Vol 6, Issue 1

2008-05-29T07:46:04Z

ddc-users-request@... ?????:

> Send ddc-users mailing list submissions to
> ddc-users@...
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://groupman.bbaw.de/mailman/listinfo/ddc-users
> or, via email, send a message with subject or body 'help' to
> ddc-users-request@...
>
> You can reach the person managing the list at
> ddc-users-owner@...
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ddc-users digest..."
>
>
> Today's Topics:
>
> 1. Case-insensitive matching in RE (tobias roth)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 27 May 2008 17:11:54 +0200
> From: tobias roth <tobias.roth@...>
> Subject: [ddc-users] Case-insensitive matching in RE
> To: ddc-users@...
> Message-ID: <200805271711.54674.tobias.roth@...>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi,
>
> I wanted to do (pseudo-)case-insensitive matching on a bibliographical
> field in DDC (v. 1.74-1, python-client):
>
> etwas #has_field[author,/\b[mM]eier\b/]
>
> hits: 0
>
> Square brackets always seem to signal the end of the has_field option - or
> can you escape them?
> Or is there any other way to do or simulate case-insensitive matching in a
> regex?
>
> Cheers,
> tobi
>

I think, it is a bug, there is no way to use brackets inside #has_field.
I should fix it tt in the next version.

Best regards,
Alexey Sokirko
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Case-insensitive matching in RE

2008-05-27T08:11:54Z

Hi,

I wanted to do (pseudo-)case-insensitive matching on a bibliographical
field in DDC (v. 1.74-1, python-client):

etwas #has_field[author,/\b[mM]eier\b/]

hits: 0

Square brackets always seem to signal the end of the has_field option - or
can you escape them?
Or is there any other way to do or simulate case-insensitive matching in a
regex?

Cheers,
tobi
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: Re: ddc-users Digest, Vol 5, Issue 1

2008-03-12T06:39:21Z

Hi Alexey,

On Mar 12, 2008, sokirko@... wrote:

> Today's Topics:
>
> 1. sort by l/r context (Matej Durco)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 10 Mar 2008 13:43:26 +0100
> From: Matej Durco <xnrn@...>
> Subject: [ddc-users] sort by l/r context
> To: ddc-users@...
> Message-ID: <47D52CEE.6090507@...>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> hi,
>
> i'd like to know what is the current status
> of the feature: sorting hits by left/right context
>
> what is the planned query-syntax for this feature?
>
> (and: how is handled the case, when there are more matches in one hit?)
>
> regards
> matej
>
>
I thought this feature was implemeted. Kai, have you tested it with DWDS
project?

I already answered Matej via pm. We tested it on a small corpus and it seems to work. But i haven't released any binaries so far. Right now we're trying to integrate our own TAGH morphology into ddc. Once we're done with that, i'm going to release the new version.

As far as i understood Matej, he wants to be able to display and sort each match within each sentence. As far as i know this is impossible in ddc.

Alex Siebert has some questions concerning ddc. I don't dare to translate, because i'm confused about his questions:

"Vlt. kannst du ihn auch noch fragen, wie man aus der Lemmaspalte die Token als Sortieroption nutzen kann.

So dass es mit Token und Lemma spalte funktioniert.

 Eigentlich wäre es schön, wenn es mit jeder Spalte in der Abfrage funktioniert also auch mit

"$p=/VV/ #links" ?

  so:  ConcordConsole  QUERY :: /die/ #left

 ReadVectorInner:length_error exception is caught: vector::reserve terminate called after throwing an instance of 'std::length_error'   what():  vector::reserve Abgebrochen

"/die/ #right" geht!

  ca. 52.000 Dateien  Bei folgendem Query geht er auch hops: $p = /VVFI/ #left Speicherzugriffsfehler

Die opt Datei:

-----------------------------------------------------------------------------------

German

 IndexType Free_Index

 Indices [Token w normal]; [Lemma l normal storage]; [Pos p normal]; [Thes t normal]; [Id i normal]; [Fu f normal] HitBorders [s:sentence:default]; [c:clause] Bigrams  Token 2 s Bigrams  Lemma 2 s OutputBibliographyOfHits ResumeOnIndexErrors TfIdfRank    2.0 NearRank     1.0 PositionRank 1.0 Bibl string 1 textClass /TEI.2/teiHeader/profileDesc/textClass/keywords/term Bibl string 1 title /TEI.2/teiHeader/fileDesc/titleStmt/title[@type="main"] Bibl integer 0 date /TEI.2/teiHeader/fileDesc/sourceDesc[@id="orig"]/biblFull/publicationStmt/date[@id="first"] Bibl string 1 avail /TEI.2/teiHeader/fileDesc/publicationStmt/availability/@n Bibl string 0 orig /TEI.2/teiHeader/fileDesc/sourceDesc[@id="orig"]/bibl Bibl integer 1 page /TEI.2/text/body/pb/@n Bibl string 1 idno /TEI.2/teiHeader/fileDesc/publicationStmt/idno Bibl string 0 author /TEI.2/teiHeader/fileDesc/sourceDesc[@id="orig"]/biblFull/titleStmt/author Bibl integer 0 extent /TEI.2/teiHeader/fileDesc/extent Bibl integer 0 random /TEI.2/teiHeader/fileDesc/random Bibl string 0 randstr /TEI.2/teiHeader/fileDesc/random Textarea body /TEI.2/text/body IndicesToShow 1 2 InterpDelimiter ^

Best regards,

Kai

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: ddc-users Digest, Vol 5, Issue 1

2008-03-12T06:01:16Z

> Today's Topics:
>
> 1. sort by l/r context (Matej Durco)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 10 Mar 2008 13:43:26 +0100
> From: Matej Durco <xnrn@...>
> Subject: [ddc-users] sort by l/r context
> To: ddc-users@...
> Message-ID: <47D52CEE.6090507@...>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> hi,
>
> i'd like to know what is the current status
> of the feature: sorting hits by left/right context
>
> what is the planned query-syntax for this feature?
>
> (and: how is handled the case, when there are more matches in one hit?)
>
> regards
> matej
>
>

I thought this feature was implemeted. Kai, have you tested it with DWDS
project?

Best regards,
Alexey Sokirko
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

sort by l/r context

2008-03-10T05:43:26Z

hi,

i'd like to know what is the current status
of the feature: sorting hits by left/right context

what is the planned query-syntax for this feature?

(and: how is handled the case, when there are more matches in one hit?)

regards
matej
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

DDC-PHP 0.02 release

2008-01-10T09:20:07Z

Hello,

the ddc-concordance team is pleased to announce the release of ddc-php0.02.
Basically it's a client script in php which can access a (remote) ddc
server.

It can be downloaded from
>http://downloads.sourceforge.net/ddc-concordance/ddc-php-client-0.02.tgz?use_mirror=osdn

If you find a bug in this release, please submit a bug report on this
mailing list.
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: integrate ddc in web-site

2007-07-11T06:46:21Z

Hi Matej,

currently we use the command-line client ddc_xml in our web-site. But
there are some problems with this setup:
- calling shell-based applications is generally dangerous, especially
concerning injection of command line code (similar to sql injection).
Therfore it is necessary to wrap queries within files (using the -f
option), which will not be expanded by the shell.
- creating those files takes some time on the server side especially
if you have to use multiple queries. You can minimize this problem
using ramdisks.

Afterall, this solution worked great for several years, but now we
decided to communicate directly via socket with the ddc server, which
should also be slightly faster. I.e. i'm going to integrate the python-
client within our zope/plone setup. The software we're going to publish
for the c4 will be based on this socket solution.

Anyway, the output from either ddc_xml or the socket way is plain text
only. The text output has some delimiters inside which are used by the
split function to divide up the text into metadata and hits. DDC again
marks hits with special characters which are translated by zope/plone
into html markup (<b> and </b>) to do some highlighting of the hits
(like google). The other stuff (page layout, graphics) is inserted via
zope templates, which keeps the code clean. Zope/Plone have Content
Management functions available, so you can easily edit static pages.

Basically, that's how it works. Just ask me if you have any further
questions/suggestions.

regards,
Kai

>----Original Message----
>From: xnrn@...
>Date: Jul 11, 2007 12:43
>To: <ddc-users@...>
>Subj: [ddc-users] integrate ddc in web-site
>
>hi,
>
>this is question especially for people in dwds-berlin:
>
>How do you integrate ddc in the web-site?
>Do you use the cgi-app Search?
>Or similarily to what is used in the python-sample
>ie communicating via sockets directly
>from the web-app logic to the server?
>
>In other words how is the collaboration/configuration
>between the web-app and ddc designed/layouted
>which allows sending the user a query and getting the formatted html

>the result?
>
>Thanks
>
>Regards
>
>Matej
>_______________________________________________
>ddc-users mailing list
>ddc-users@...
>http://groupman.bbaw.de/mailman/listinfo/ddc-users
>

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

integrate ddc in web-site

2007-07-11T04:43:30Z

hi,

this is question especially for people in dwds-berlin:

How do you integrate ddc in the web-site?
Do you use the cgi-app Search?
Or similarily to what is used in the python-sample
ie communicating via sockets directly
from the web-app logic to the server?

In other words how is the collaboration/configuration
between the web-app and ddc designed/layouted
which allows sending the user a query and getting the formatted html as
the result?

Thanks

Regards

Matej
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

Re: ddc-users Digest, Vol 3, Issue 1

2007-07-05T23:56:56Z

ddc-users-request@... ?????:

> Send ddc-users mailing list submissions to
> ddc-users@...
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://groupman.bbaw.de/mailman/listinfo/ddc-users
> or, via email, send a message with subject or body 'help' to
> ddc-users-request@...
>
> You can reach the person managing the list at
> ddc-users-owner@...
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ddc-users digest..."
>
>
> Today's Topics:
>
> 1. threading in ddc? (Matej Durco)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 04 Jul 2007 15:32:07 +0200
> From: Matej Durco <xnrn@...>
> Subject: [ddc-users] threading in ddc?
> To: ddc-users@...
> Message-ID: <468BA157.7010201@...>
> Content-Type: text/plain; charset=ISO-8859-15; format=flowed
>
> hi,
>
> i'd like to understand, how threading is implemented in ConcordDaemon.
>
> if I understand correctly:
> Daemon as Server
> sends requests and receives results
> to the Threads as PossibleHosts
> in Corpora.cpp-GetFirstHitsFromCorpora
> (in 2 loops):
>
> Ln 407
> string S = Format ("get_first_hits %s\x1%i %i", Query.c_str(), TimeOut,
> ResultLimit);
> string ErrorStr;
> if (!SendString(Result.m_Sockets[i], S.c_str(), S.length(), ErrorStr))
>
> Ln 425
> Result.m_NetworkError = RecieveString(Result.m_Sockets[i], S, TimeOut);
>
> But he actually receives the Results one by one, is that correct?
>
>

Yes it is true, DDC Server waits for all results, and recieves them one
by one. Accumulating all results is necessary for general query
statistics like the number of
sentences that contain the given query. Recieving the results one by one
could be optimized by moving all recieving opearations to different
threads. But it is not the speed bottleneck, the speed bottleneck is in
reading word occurrences from the disk.

Best regards,
Alexey Sokirko

>
> ------------------------------
>
> _______________________________________________
> ddc-users mailing list
> ddc-users@...
> http://groupman.bbaw.de/mailman/listinfo/ddc-users
>
>
> End of ddc-users Digest, Vol 3, Issue 1
> ***************************************
>
>

_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

threading in ddc?

2007-07-04T07:32:07Z

hi,

i'd like to understand, how threading is implemented in ConcordDaemon.

if I understand correctly:
Daemon as Server
sends requests and receives results
to the Threads as PossibleHosts
in Corpora.cpp-GetFirstHitsFromCorpora
(in 2 loops):

Ln 407
string S = Format ("get_first_hits %s\x1%i %i", Query.c_str(), TimeOut,
ResultLimit);
string ErrorStr;
if (!SendString(Result.m_Sockets[i], S.c_str(), S.length(), ErrorStr))

Ln 425
Result.m_NetworkError = RecieveString(Result.m_Sockets[i], S, TimeOut);

But he actually receives the Results one by one, is that correct?

Thanks

Regards

Matej
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users

DDC-Python 0.01 release.

2007-06-07T09:04:24Z

Hello,

the ddc-concordance team is pleased to announce the release of ddc-
python 0.01.
Basically it's a small client script in python which can access a
(remote) ddc server.

It can be downloaded from http://sourceforge.net/project/showfiles.php?
group_id=97101&package_id=234602

If you find a bug in this release, please submit a bug report on this
mailing list.
_______________________________________________
ddc-users mailing list
ddc-users@...
http://groupman.bbaw.de/mailman/listinfo/ddc-users