How to handle case where a keyword can also be a function call?

[Nathan Ward]

I'm making a parser for a language that uses "long" as a data type and also has a function by the same name that takes a string as a parameter and converts it to a long value. How can I setup the javacc grammar to distinguish between the token of "long" and an identifier that is the name of a function in a "FunctionCall" production?

 

TOKEN :

{ < LONG: "long" > }

void FunctionCall() :

{ }

{ Variable() "(" [Parameters()] ")" }

Target language statement that is being matched as the token "long", but should be treated as a FunctionCall():

 

long(ls_id)

 

 

Nathan Ward

ResQSoft, Inc.

703.861.9103

www.resqsoft.com

 

[Laughing Man]

Some parts of this message have been removed. Learn more about Nabble's security policy.

I'm not sure exactly how to do it, but checking for an opening parenthesis seems like it might be the way to go.

---

From: nward@...
To: users@...
Date: Wed, 16 Jul 2008 14:15:59 -0400
Subject: [JavaCC] How to handle case where a keyword can also be a function call?

I'm making a parser for a language that uses "long" as a data type and also has a function by the same name that takes a string as a parameter and converts it to a long value. How can I setup the javacc grammar to distinguish between the token of "long" and an identifier that is the name of a function in a "FunctionCall" production?

 

TOKEN :

{ < LONG: "long" > }

void FunctionCall() :

{ }

{ Variable() "(" [Parameters()] ")" }

Target language statement that is being matched as the token "long", but should be treated as a FunctionCall():

 

long(ls_id)

 

 

Nathan Ward

ResQSoft, Inc.

703.861.9103

www.resqsoft.com

 

---

Connect to the next generation of MSN Messenger  Get it now!

[Nathan Ward]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Thanks, but I believe what is happening is that the tokenize identifies it as a token so that the parser doesn't get a chance to evaluate the production that would check for an open parenthesis. Maybe I can refractor the grammar to not define "long" and other data type keywords as tokens.

 

   Nathan

 

---

From: Laughing Man [mailto:xuincherguixe@...]
Sent: Wednesday, July 16, 2008 2:30 PM
To: users@...
Subject: RE: [JavaCC] How to handle case where a keyword can also be a function call?

I'm not sure exactly how to do it, but checking for an opening parenthesis seems like it might be the way to go.

---

From: nward@...
To: users@...
Date: Wed, 16 Jul 2008 14:15:59 -0400
Subject: [JavaCC] How to handle case where a keyword can also be a function call?

I'm making a parser for a language that uses "long" as a data type and also has a function by the same name that takes a string as a parameter and converts it to a long value. How can I setup the javacc grammar to distinguish between the token of "long" and an identifier that is the name of a function in a "FunctionCall" production?

 

TOKEN :

{ < LONG: "long" > }

void FunctionCall() :

{ }

{ Variable() "(" [Parameters()] ")" }

Target language statement that is being matched as the token "long", but should be treated as a FunctionCall():

 

long(ls_id)

 

 

Nathan Ward

ResQSoft, Inc.

703.861.9103

www.resqsoft.com

 

---

Connect to the next generation of MSN Messenger  Get it now!

[darose]

Nathan Ward wrote:

> I'm making a parser for a language that uses "long" as a data type and also
> has a function by the same name that takes a string as a parameter and
> converts it to a long value. How can I setup the javacc grammar to
> distinguish between the token of "long" and an identifier that is the name
> of a function in a "FunctionCall" production?
>  
> TOKEN :
>
> { < LONG: "long" > }
>
> void FunctionCall() :
>
> { }
>
> { Variable() "(" [Parameters()] ")" }
>
> Target language statement that is being matched as the token "long", but
> should be treated as a FunctionCall():
>  
> long(ls_id)
>  
>  
> Nathan Ward

Use 2 different lexical states?

DR

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[J.Chris Findlay]

The way most grammars seem to do this is special-case the cast-functions, so you have as you do a definition for FunctionCall(), but you also have an almost identical one where Variable() is replaced with TypeName() (which includes the token for "long").
Or replace Variable() with VariableOrType() which can accept both, and is only used in places where that makes sense.

Otherwise you end up stuck with the fact that the 4 chars of "long" are always one or the other of TypeName or GeneralIdentifier.
It is usually preferable to have type names stay as type tokens, rather than have everything be just identifiers (unless you have no built-in types), but this means the grammar must accept them wherever they can validly show up.

On Thu, Jul 17, 2008 at 6:15 AM, Nathan Ward <nward@...> wrote:

I'm making a parser for a language that uses "long" as a data type and also has a function by the same name that takes a string as a parameter and converts it to a long value. How can I setup the javacc grammar to distinguish between the token of "long" and an identifier that is the name of a function in a "FunctionCall" production?

 

TOKEN

:

{ < LONG: "long" > }

void FunctionCall() :

{ }

{ Variable() "(" [Parameters()] ")" }

Target language statement that is being matched as the token "long", but should be treated as a FunctionCall():

 

long(ls_id)

 

 

Nathan Ward

ResQSoft, Inc.

703.861.9103

www.resqsoft.com

 

--
- J.Chris Findlay
(c:

[Stuart A. Yeates]

In languages such as Perl, you can't know the correct parsing until
runtime, because you can include files based on temporally varying
function calls and the content of those files can determine whether
strings are functions, variables or literals. Thus the need for the -w
option.

So the answer to the original question is "it depends."

cheers
stuart

On Thu, Jul 17, 2008 at 9:21 AM, J.Chris Findlay
<j.chris.findlay@...> wrote:

> The way most grammars seem to do this is special-case the cast-functions, so
> you have as you do a definition for FunctionCall(), but you also have an
> almost identical one where Variable() is replaced with TypeName() (which
> includes the token for "long").
> Or replace Variable() with VariableOrType() which can accept both, and is
> only used in places where that makes sense.
>
> Otherwise you end up stuck with the fact that the 4 chars of "long" are
> always one or the other of TypeName or GeneralIdentifier.
> It is usually preferable to have type names stay as type tokens, rather than
> have everything be just identifiers (unless you have no built-in types), but
> this means the grammar must accept them wherever they can validly show up.
>
> On Thu, Jul 17, 2008 at 6:15 AM, Nathan Ward <nward@...> wrote:
>>
>> I'm making a parser for a language that uses "long" as a data type and
>> also has a function by the same name that takes a string as a parameter and
>> converts it to a long value. How can I setup the javacc grammar to
>> distinguish between the token of "long" and an identifier that is the name
>> of a function in a "FunctionCall" production?
>>
>>
>> TOKEN
>>
>> :
>>
>> { < LONG: "long" > }
>>
>> void FunctionCall() :
>>
>> { }
>>
>> { Variable() "(" [Parameters()] ")" }
>>
>> Target language statement that is being matched as the token "long", but
>> should be treated as a FunctionCall():
>>
>> long(ls_id)
>>
>>
>> Nathan Ward
>> ResQSoft, Inc.
>> 703.861.9103
>> www.resqsoft.com
>>
>
>
> --
> - J.Chris Findlay
> (c:

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Randall Schulz]

On Wednesday 16 July 2008 11:15, Nathan Ward wrote:
> I'm making a parser for a language that uses "long" as a data type
> and also has a function by the same name that takes a string as a
> parameter and converts it to a long value. How can I setup the javacc
> grammar to distinguish between the token of "long" and an identifier
> that is the name of a function in a "FunctionCall" production?

I've faced similar requirements. What I usually do is create a
production that matches the generic name / label / symbol token plus
the enumeration of all the keywords that are not reserved in the
context of that production. Then the upper levels of the grammar use
the generic name / label /symbol token in places where the grammar's
keywords are reserved and uses the (or one of the) name + keyword
productions in context where that's what required.

So if you have (not JavaCC, just a generic pseudo-EBNF notation)

/* Numeric type keywords: */
integer: "int" ;
long: "long" ;
float: "float" ;
double: "double" ;

/* Control keywords: */
if: "if" ;
while: "while" ;
switch: "switch" ;

/* Generic names: */
name: [A-Z_a-z][A-Z_a-z0-9]* ;

/* Function names (coercions or user-defined): */
function: ( integer | long | float | double | name ) ;

/* Variable names: */
variable: name ;


This may or may not lead to other ambiguities in your grammar, but it
should be possible to handle them without too much trouble.


Randall Schulz

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Nathan Ward]

> Use 2 different lexical states?

But, I don't have anything to key on in order to switch lexical states.  

-----Original Message-----
From: David Rosenstrauch [mailto:darose@...]
Sent: Wednesday, July 16, 2008 3:26 PM
To: users@...
Subject: Re: [JavaCC] How to handle case where a keyword can also be a
function call?

Nathan Ward wrote:

> I'm making a parser for a language that uses "long" as a data type and
> also has a function by the same name that takes a string as a
> parameter and converts it to a long value. How can I setup the javacc
> grammar to distinguish between the token of "long" and an identifier
> that is the name of a function in a "FunctionCall" production?
>  
> TOKEN :
>
> { < LONG: "long" > }
>
> void FunctionCall() :
>
> { }
>
> { Variable() "(" [Parameters()] ")" }
>
> Target language statement that is being matched as the token "long",
> but should be treated as a FunctionCall():
>  
> long(ls_id)
>  
>  
> Nathan Ward

Use 2 different lexical states?

DR

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Nathan Ward]

Thanks! That sounds reasonable.

 

It seems that I don't actually have the problem that I thought I had. I inherited this grammar from someone who has left the company and I don't completely understand it or JavaCC yet.

 

The token "LONG" is already used in two productions. So, that apparently is working fine. It was a matter of using one of those in the production in question (the FunctionCall() production) in an alternative with Variable(), i.e. (StandardFunction() | Variable()). I think this is basically what you were suggestiong.

 

   Nathan

---

From: J.Chris Findlay [mailto:j.chris.findlay@...]
Sent: Wednesday, July 16, 2008 5:22 PM
To: users@...
Subject: Re: [JavaCC] How to handle case where a keyword can also be a function call?

The way most grammars seem to do this is special-case the cast-functions, so you have as you do a definition for FunctionCall(), but you also have an almost identical one where Variable() is replaced with TypeName() (which includes the token for "long").
Or replace Variable() with VariableOrType() which can accept both, and is only used in places where that makes sense.

Otherwise you end up stuck with the fact that the 4 chars of "long" are always one or the other of TypeName or GeneralIdentifier.
It is usually preferable to have type names stay as type tokens, rather than have everything be just identifiers (unless you have no built-in types), but this means the grammar must accept them wherever they can validly show up.

On Thu, Jul 17, 2008 at 6:15 AM, Nathan Ward <nward@...> wrote:

I'm making a parser for a language that uses "long" as a data type and also has a function by the same name that takes a string as a parameter and converts it to a long value. How can I setup the javacc grammar to distinguish between the token of "long" and an identifier that is the name of a function in a "FunctionCall" production?

 

TOKEN

:

{ < LONG: "long" > }

void FunctionCall() :

{ }

{ Variable() "(" [Parameters()] ")" }

Target language statement that is being matched as the token "long", but should be treated as a FunctionCall():

 

long(ls_id)

 

 

Nathan Ward

ResQSoft, Inc.

703.861.9103

www.resqsoft.com

 

--
- J.Chris Findlay
(c:

[darose]

Are you certain?  I don't know your grammar, but it sounded to me like
it's not valid syntax to use a function name when the parser is
expecting a data type, and vice versa.  If that's the case, then
wouldn't you be able to switch lexical states, thereby setting the
parser to correctly expect the next token type?

DR

Nathan Ward wrote:

>> Use 2 different lexical states?
>
> But, I don't have anything to key on in order to switch lexical states.  
>
> -----Original Message-----
> From: David Rosenstrauch [mailto:darose@...]
> Sent: Wednesday, July 16, 2008 3:26 PM
> To: users@...
> Subject: Re: [JavaCC] How to handle case where a keyword can also be a
> function call?
>
> Nathan Ward wrote:
>> I'm making a parser for a language that uses "long" as a data type and
>> also has a function by the same name that takes a string as a
>> parameter and converts it to a long value. How can I setup the javacc
>> grammar to distinguish between the token of "long" and an identifier
>> that is the name of a function in a "FunctionCall" production?
>>  
>> TOKEN :
>>
>> { < LONG: "long" > }
>>
>> void FunctionCall() :
>>
>> { }
>>
>> { Variable() "(" [Parameters()] ")" }
>>
>> Target language statement that is being matched as the token "long",
>> but should be treated as a FunctionCall():
>>  
>> long(ls_id)
>>  
>>  
>> Nathan Ward
>
> Use 2 different lexical states?
>
> DR

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Theodore Norvell]

Nathan Ward wrote:
> Thanks! That sounds reasonable.
I think Chris nailed this best way.  Note that the FAQ outlines three
approaches to the problem.
This is one.  See
http://www.engr.mun.ca/~theo/JavaCC-FAQ/javacc-faq-moz.htm#tth_sEc4.19 .

Cheers,
Theo

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Nathan Ward]

Thanks Theo! Very helpful. -- Nathan

-----Original Message-----
From: Theodore Norvell [mailto:theo@...]
Sent: Monday, July 21, 2008 11:04 AM
To: users@...
Subject: Re: [JavaCC] How to handle case where a keyword can also be a
function call?

Nathan Ward wrote:
> Thanks! That sounds reasonable.
I think Chris nailed this best way.  Note that the FAQ outlines three
approaches to the problem.
This is one.  See
http://www.engr.mun.ca/~theo/JavaCC-FAQ/javacc-faq-moz.htm#tth_sEc4.19 .

Cheers,
Theo

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...