I'm not 100% sure that this is the right forum, but I don't see a
javacc.devel list, just this user list. (There are the automated
notification lists for CVS and the bugtracker issues, but I can't really
see starting a discussion on those...)
I'm a long-time user of javacc. The main, fairly well known thing that I
have used it for is to generate the parser for FreeMarker, a well known
java template library. I've never had much cause to get involved in this
community, since I was always able to use figure out what I needed to do
with JavaCC. It just worked. However, for various reasons, I have
recently been looking at ANTLR and took a run at that. And I ended up
looking more closely at the tools and how they do things and getting
interested in the whole subject.
Anyway, over the weekend, I was playing around with the code for the
jjdoc tools. Now, I'll be frank. I don't think jjdoc is nearly as useful
a tool as it potentially could be. The reason is mainly that it is very
hard to customize the output. Basically, to put it bluntly, the existing
jjdoc code uses the antipattern of generating HTML with println
statements, so, to all intents and purposes, the output is not
customizable.
So, I took a look at how to do something about this, not so very much
because I am interested in jjdoc per se. Largely I saw jjdoc as a way of
getting involved, mucking with the code in a way that presents no
technical risk really, but it allows me to do some useful things while
I'm still getting to know the internals.
So, basically, I created a new org.javacc.jjdoc.Generator implementation
called TemplateBasedGenerator that leverages the FreeMarker template
library. (Quite fitting, since FreeMarker itself uses JavaCC to generate
its parser...) And I wrote a template that produces similar output to
what the existing HTMLGenerator produces. It's not exactly the same, but
the template could be tweaked to produce exactly the same output. But
that is actually not very important. The important aspect of this is
that you have a template that you can edit fairly easily to customize
the output to how you want it.
Anyway, I'm attaching the jjdoc.ftl template and the
TemplateBasedGenerator.java source so you can see what I'm talking
about. I really don't think this basic idea should be very
controversial. A tool like jjdoc really should be template-based.
Generating HTML markup with println statements is an antipattern.
FreeMarker and other similar template libraries were designed for this,
they are the right tool for the job.
Now, as for what I have right now, it still could be refined and
polished quite a bit, but this works. I had to change two spots in the
code to support some things I did, but it was very very minor.
Basically, the java source implements freemarker.template.ObjectWrapper
interface and contains the basic logic to present the internal JavaCC
objects to the template layer. And then you have these helper objects
like ExpansionModel and ProductionModel and so on that exist to expose
the information that the template layer needs to generate the output.
And it really does add value. A sophisticated user who wants to generate
different useful views of their parser logic could work up various
templates that, based on the same grammar, generate different navigable
views on it. And, of course, those templates could be shared among the
community and so on.
<#escape x as (x)?html>
<#if css_file?has_content>
<#if input_file?has_content>
BNF for ${input_file}
<#set title = "BNF for " + input_file>
<#else>
A BNF grammar by JJDoc
<#set title = "BNF Grammar">
<#macro show_expansion expansion>
<#if expansion.type = "choice">
<#list expansion.choices as choice>
<@show_subexpansion choice/>
<#if choice_has_next>|
<#else>
<@show_subexpansion expansion/>
<#macro show_subexpansion expansion>
<#switch expansion.type>
<#case "Action"><@show_action expansion/><#break>
<#case "Choice"><@show_choice expansion/><#break>
<#case "NonTerminal"><@show_nonterminal expansion/><#break>
<#case "Lookahead"><@show_lookahead expansion/><#break>
<#case "OneOrMore"><@show_one_or_more expansion/><#break>
<#case "RegularExpression"><@show_regular_expression expansion/><#break>
<#case "Sequence"><@show_sequence expansion/><#break>
<#case "TryBlock"><@show_try_block expansion/><#break>
<#case "ZeroOrMore"><@show_zero_or_more expansion/><#break>
<#case "ZeroOrOne"><@show_zero_or_one expansion/><#break>
<#default><#stop "Unknown expansion type: " + expansion.type>
<#macro show_sequence sequence>
<#list sequence.units as unit>
<#var needsParenthesis = unit.type == "sequence" || unit.type == "choice">
<#if needsParenthesis>(
<@show_subexpansion unit/>
<#if needsParenthesis>)
<#macro show_choice choice>
<#list choice.choices as sub>
<@show_subexpansion sub/>
<#if sub_has_next>|
<#macro show_nonterminal nt>
${nt.name}
<#macro show_zero_or_more zm>
(<@show_subexpansion zm.nested/>)*
<#macro show_zero_or_one zo>
(<@show_subexpansion zo.nested/>)?
<#macro show_one_or_more om>
(<@show_subexpansion om.nested/>)+
<#macro show_regular_expression regex>${regex}
<#macro show_action action>
<#macro show_lookahead lookahead>
<#macro show_try_block try_block>
/* Copyright (c) 2008, Sun Microsystems, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of the Sun Microsystems, Inc. nor the names of its
* contributors may be used to endorse or promote products derived from
* this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
* THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
* Output BNF using a FreeMarker template
*/
public class TemplateBasedGenerator
extends TextGenerator
implements ObjectWrapper, JavaCCParserConstants {
private Configuration fmConfig;
private Template template;
private SimpleHash dataModel;
private Map<String, String> id_map = new HashMap<String, String>();
private int id=1;
private ObjectWrapper fallbackWrapper = new DefaultObjectWrapper();
public TemplateBasedGenerator() {
this.fmConfig = new Configuration();
dataModel = new SimpleHash(this);
fmConfig.setObjectWrapper(this);
File currentDir = new File(".").getAbsoluteFile();
TemplateLoader loader1 = null;
TemplateLoader loader2 = new ClassTemplateLoader(this.getClass(), "");
try {
loader1 = new FileTemplateLoader(currentDir);
fmConfig.setTemplateLoader(new MultiTemplateLoader(new TemplateLoader[] {
loader1, loader2 }));
} catch (IOException ioe) {
debug("Cannot load from file system, falling back on default template");
fmConfig.setTemplateLoader(new MultiTemplateLoader(
new TemplateLoader[] { loader2 }));
}
}
public TemplateModel wrap(Object obj) throws TemplateModelException {
if (obj instanceof TemplateModel) {
return (TemplateModel) obj;
}
if (obj instanceof List) {
return new SimpleSequence((List) obj, this);
}
if (obj instanceof Map) {
return new SimpleHash((Map)obj, this);
}
if (obj instanceof Token) {
return new TokenModel((Token) obj);
}
if (obj instanceof NormalProduction) {
return new ProductionModel((NormalProduction) obj);
}
if (obj instanceof Expansion) {
return new ExpansionModel((Expansion) obj);
}
return fallbackWrapper.wrap(obj);
}
public void documentStart() {
try {
this.template = fmConfig.getTemplate("/src/org/javacc/jjdoc/jjdoc.ftl");
} catch (Exception e) {
debug("Failed to load template");
if (e instanceof RuntimeException)
throw (RuntimeException) e;
throw new RuntimeException(e);
}
dataModel.put("input_file", JJDocGlobals.input_file);
dataModel.put("css_file", JJDocOptions.getCSS());
dataModel.put("nonterminals", JavaCCGlobals.bnfproductions);
dataModel.put("token_productions", JavaCCGlobals.rexprlist);
dataModel.put("get_id", new GetIDMethod());
ostr = new PrintWriter(new Writer() {
public void flush() {}
public void close() {}
public void write(char[] chars, int i, int j){}
});
}
public void documentEnd() {
ostr = create_output_stream();
try {
template.process(dataModel, ostr);
ostr.close();
} catch (Exception e) {
debug(e.getMessage());
if (e instanceof RuntimeException)
throw (RuntimeException) e;
throw new RuntimeException(e);
}
}
class TokenModel implements TemplateHashModel, TemplateScalarModel {
final Token token;
TokenModel(Token token) {
this.token = token;
}
public String getAsString() {
return token.image;
}
public TemplateModel get(String key) throws TemplateModelException {
if (key.equals("kind")) {
String result = tokenImage[token.kind];
result = result.substring(1, result.length() -1);
return wrap(result);
}
throw new TemplateModelException("Unknown key: " + key);
}
public boolean isEmpty() {
return false;
}
}
class ProductionModel implements TemplateHashModel {
final NormalProduction production;
if (key.equals("nested")) {
Expansion nested = null;
if (expansion instanceof ZeroOrOne) nested = ((ZeroOrOne) expansion).expansion;
else if (expansion instanceof ZeroOrMore) nested = ((ZeroOrMore) expansion).expansion;
else if (expansion instanceof OneOrMore) nested = ((OneOrMore) expansion).expansion;
return wrap(nested);
}
if (key.equals("type")) {
String s=null;
if (expansion instanceof RegularExpression) s = "RegularExpression";
else {
s = expansion.getClass().getName();
s = s.substring(s.lastIndexOf('.')+1);
}
return wrap(s);
}
throw new TemplateModelException("Unknown key: " + key);
}
public boolean isEmpty() {return false;}
}
class GetIDMethod implements TemplateMethodModel {
public Object exec(List args) {
String arg = args.get(0).toString();
String i = id_map.get(arg);
if (i == null) {
i = "prod" + id++;
id_map.put(arg, i);
}
return i;
}
}
static public String mungeFormalComment(String formalComment) {
if (!formalComment.startsWith("/**")) return formalComment;
formalComment = formalComment.substring(3, formalComment.length() -2);
StringBuilder buf = new StringBuilder();
StringTokenizer st = new StringTokenizer(formalComment, "\n\r");
while (st.hasMoreTokens()) {
String tok = st.nextToken();
tok = tok.trim();
if (tok.startsWith("*")) tok = tok.substring(1);
if (buf.length() >0) buf.append(" ");
buf.append(tok);
}
return buf.toString();
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...
Re: Contribution and question about getting involved in the project.
Some parts of this message have been removed.
Learn more about Nabble's security policy.
Sounds really cool, however, the HTML-like template got soaked up into the email content rather than showing up as a separate attachment. Perhaps an archive of the files would work better...
I'm not 100% sure that this is the right forum, but I don't see a javacc.devel list, just this user list. (There are the automated notification lists for CVS and the bugtracker issues, but I can't really see starting a discussion on those...)
I'm a long-time user of javacc. The main, fairly well known thing that I have used it for is to generate the parser for FreeMarker, a well known java template library. I've never had much cause to get involved in this community, since I was always able to use figure out what I needed to do with JavaCC. It just worked. However, for various reasons, I have recently been looking at ANTLR and took a run at that. And I ended up looking more closely at the tools and how they do things and getting interested in the whole subject.
Anyway, over the weekend, I was playing around with the code for the jjdoc tools. Now, I'll be frank. I don't think jjdoc is nearly as useful a tool as it potentially could be. The reason is mainly that it is very hard to customize the output. Basically, to put it bluntly, the existing jjdoc code uses the antipattern of generating HTML with println statements, so, to all intents and purposes, the output is not customizable.
So, I took a look at how to do something about this, not so very much because I am interested in jjdoc per se. Largely I saw jjdoc as a way of getting involved, mucking with the code in a way that presents no technical risk really, but it allows me to do some useful things while I'm still getting to know the internals.
So, basically, I created a new org.javacc.jjdoc.Generator implementation
called TemplateBasedGenerator that leverages the FreeMarker template
library. (Quite fitting, since FreeMarker itself uses JavaCC to generate its parser...) And I wrote a template that produces similar output to what the existing HTMLGenerator produces. It's not exactly the same, but the template could be tweaked to produce exactly the same output. But that is actually not very important. The important aspect of this is that you have a template that you can edit fairly easily to customize the output to how you want it.
Anyway, I'm attaching the jjdoc.ftl template and the
TemplateBasedGenerator.java source so you can see what I'm talking
about. I really don't think this basic idea should be very controversial. A tool like jjdoc really should be template-based. Generating HTML markup with println statements is an antipattern. FreeMarker and other similar template libraries were designed for this, they are the right tool for the job.
Now, as for what I have right now, it still could be refined and polished quite a bit, but this works. I had to change two spots in the code to support some things I did, but it was very very minor. Basically, the java source implements freemarker.template.ObjectWrapper interface and contains the basic logic to present the internal JavaCC objects to the template layer. And then you have these helper objects like ExpansionModel and ProductionModel and so on that exist to expose the information that the template layer needs to generate the output.
And it really does add value. A sophisticated user who wants to generate different useful views of their parser logic could work up various templates that, based on the same grammar, generate different navigable views on it. And, of course, those templates could be shared among the community and so on.
<#escape x as (x)?html>
<#if css_file?has_content>
<#if input_file?has_content>
<#set title = "BNF for " + input_file>
<#else>
<#set title = "BNF Grammar">
<#macro show_expansion expansion>
<#if expansion.type = "choice">
<#list expansion.choices as choice>
<@show_subexpansion choice/>
<#if choice_has_next>|
<#else>
<@show_subexpansion expansion/>
<#macro show_subexpansion expansion>
<#switch expansion.type>
<#case "Action"><@show_action expansion/><#break>
<#case "Choice"><@show_choice expansion/><#break>
<#case "NonTerminal"><@show_nonterminal expansion/><#break>
<#case "Lookahead"><@show_lookahead expansion/><#break>
<#case "OneOrMore"><@show_one_or_more expansion/><#break>
<#case "RegularExpression"><@show_regular_expression expansion/><#break>
<#case "Sequence"><@show_sequence expansion/><#break>
<#case "TryBlock"><@show_try_block expansion/><#break>
<#case "ZeroOrMore"><@show_zero_or_more expansion/><#break>
<#case "ZeroOrOne"><@show_zero_or_one expansion/><#break>
<#default><#stop "Unknown expansion type: " + expansion.type>
<#macro show_sequence sequence>
<#list sequence.units as unit>
<#var needsParenthesis = unit.type == "sequence" || unit.type == "choice">
<#if needsParenthesis>(
<@show_subexpansion unit/>
<#if needsParenthesis>)
<#macro show_choice choice>
<#list choice.choices as sub>
<@show_subexpansion sub/>
<#if sub_has_next>|
<#macro show_nonterminal nt>
${nt.name}
<#macro show_zero_or_more zm>
(<@show_subexpansion zm.nested/>)*
<#macro show_zero_or_one zo>
(<@show_subexpansion zo.nested/>)?
<#macro show_one_or_more om>
(<@show_subexpansion om.nested/>)+
<#macro show_regular_expression regex>${regex}
<#macro show_action action>
<#macro show_lookahead lookahead>
<#macro show_try_block try_block>
/* Copyright (c) 2008, Sun Microsystems, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of the Sun Microsystems, Inc. nor the names of its
* contributors may be used to endorse or promote products derived from
* this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
* THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
* Output BNF using a FreeMarker template
*/
public class TemplateBasedGenerator
extends TextGenerator
implements ObjectWrapper, JavaCCParserConstants {
private Configuration fmConfig;
private Template template;
private SimpleHash dataModel;
private Map<String, String> id_map = new HashMap<String, String>();
private int id=1;
private ObjectWrapper fallbackWrapper = new DefaultObjectWrapper();
public TemplateBasedGenerator() {
this.fmConfig = new Configuration();
dataModel = new SimpleHash(this);
fmConfig.setObjectWrapper(this);
File currentDir = new File(".").getAbsoluteFile();
TemplateLoader loader1 = null;
TemplateLoader loader2 = new ClassTemplateLoader(this.getClass(), "");
try {
loader1 = new FileTemplateLoader(currentDir);
fmConfig.setTemplateLoader(new MultiTemplateLoader(new TemplateLoader[] {
loader1, loader2 }));
} catch (IOException ioe) {
debug("Cannot load from file system, falling back on default template");
fmConfig.setTemplateLoader(new MultiTemplateLoader(
new TemplateLoader[] { loader2 }));
}
}
public TemplateModel wrap(Object obj) throws TemplateModelException {
if (obj instanceof TemplateModel) {
return (TemplateModel) obj;
}
if (obj instanceof List) {
return new SimpleSequence((List) obj, this);
}
if (obj instanceof Map) {
return new SimpleHash((Map)obj, this);
}
if (obj instanceof Token) {
return new TokenModel((Token) obj);
}
if (obj instanceof NormalProduction) {
return new ProductionModel((NormalProduction) obj);
}
if (obj instanceof Expansion) {
return new ExpansionModel((Expansion) obj);
}
return fallbackWrapper.wrap(obj);
}
public void documentStart() {
try {
this.template = fmConfig.getTemplate("/src/org/javacc/jjdoc/jjdoc.ftl");
} catch (Exception e) {
debug("Failed to load template");
if (e instanceof RuntimeException)
throw (RuntimeException) e;
throw new RuntimeException(e);
}
dataModel.put("input_file", JJDocGlobals.input_file);
dataModel.put("css_file", JJDocOptions.getCSS());
dataModel.put("nonterminals", JavaCCGlobals.bnfproductions);
dataModel.put("token_productions", JavaCCGlobals.rexprlist);
dataModel.put("get_id", new GetIDMethod());
ostr = new PrintWriter(new Writer() {
public void flush() {}
public void close() {}
public void write(char[] chars, int i, int j){}
});
}
public void documentEnd() {
ostr = create_output_stream();
try {
template.process(dataModel, ostr);
ostr.close();
} catch (Exception e) {
debug(e.getMessage());
if (e instanceof RuntimeException)
throw (RuntimeException) e;
throw new RuntimeException(e);
}
}
class TokenModel implements TemplateHashModel, TemplateScalarModel {
final Token token;
TokenModel(Token token) {
this.token = token;
}
public String getAsString() {
return token.image;
}
public TemplateModel get(String key) throws TemplateModelException {
if (key.equals("kind")) {
String result = tokenImage[token.kind];
result = result.substring(1, result.length() -1);
return wrap(result);
}
throw new TemplateModelException("Unknown key: " + key);
}
public boolean isEmpty() {
return false;
}
}
class ProductionModel implements TemplateHashModel {
final NormalProduction production;
if (key.equals("nested")) {
Expansion nested = null;
if (expansion instanceof ZeroOrOne) nested = ((ZeroOrOne) expansion).expansion;
else if (expansion instanceof ZeroOrMore) nested = ((ZeroOrMore) expansion).expansion;
else if (expansion instanceof OneOrMore) nested = ((OneOrMore) expansion).expansion;
return wrap(nested);
}
if (key.equals("type")) {
String s=null;
if (expansion instanceof RegularExpression) s = "RegularExpression";
else {
s = expansion.getClass().getName();
s = s.substring(s.lastIndexOf('.')+1);
}
return wrap(s);
}
throw new TemplateModelException("Unknown key: " + key);
}
public boolean isEmpty() {return false;}
}
class GetIDMethod implements TemplateMethodModel {
public Object exec(List args) {
String arg = args.get(0).toString();
String i = id_map.get(arg);
if (i == null) {
i = "prod" + id++;
id_map.put(arg, i);
}
return i;
}
}
static public String mungeFormalComment(String formalComment) {
if (!formalComment.startsWith("/**")) return formalComment;
formalComment = formalComment.substring(3, formalComment.length() -2);
StringBuilder buf = new StringBuilder();
StringTokenizer st = new StringTokenizer(formalComment, "\n\r");
while (st.hasMoreTokens()) {
String tok = st.nextToken();
tok = tok.trim();
if (tok.startsWith("*")) tok = tok.substring(1);
if (buf.length() >0) buf.append(" ");
buf.append(tok);
}
return buf.toString();
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...
-- - J.Chris Findlay (c:
Re: Contribution and question about getting involved in the project.
J.Chris Findlay wrote:
> Sounds really cool, however, the HTML-like template got soaked up into
> the email content rather than showing up as a separate attachment.
> Perhaps an archive of the files would work better...
>
Okay, here are the two files zipped.
JR
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...
Jonathan Revusky wrote:
> Hi,
>
> I'm not 100% sure that this is the right forum, but I don't see a
> javacc.devel list, just this user list. (There are the automated
> notification lists for CVS and the bugtracker issues, but I can't really
> see starting a discussion on those...)
>
Hi again,
Okay, I wrote a note in which I expressed interest in doing some work in
the project and nobody has answered in 4 days. To be fair, many people
have a long weekend starting on May 1, so maybe this is an atypical 4
days. Nonetheless, it does not give me the feeling that the project is
very active. I was googling around for a roadmap or wishlist and the
most recent thing I found was a thread from 2006, and then, AFAICS,
there was no follow-through afterwards.
Well, I have been doing some work on it over the last couple of days.
What I have done mostly is I have refactored the codebase to use generic
containers. When I first imported the codebase from CVS into my eclipse
workspace, the project had over 800 warnings, mostly related to type
safety. I've now got that down to 39. And actually, the majority of
those warnings are from code that is itself _generated_ by JavaCC, so
there was no way of getting rid of those.
The lint tool in javac, i.e. compiling all the code with javac -Xlint,
now gives 22 warnings, again, mostly from code that is itself
javacc-generated. (Before, it stopped after 100 warnings.)
A lot more cleanup and refactoring is possible, but already the code is
significantly more readable. Where before, you had something like:
Vector removeList = new Vector();
now you have:
List<List<RegExprSpec>> removeList = new ArrayList<List<RegExprSpec>>()
That is actually more verbose, but it is certainly much less opaque. One
sees the data structures that are being used. But elsewhere, the code
becomes much less cluttered as a consequence. Once I generified the
code, most of the casts that were all over the place became superfluous
and could be removed. Also, I largely replaced looping constructs with
the extended for loop in Java 1.5.
So, where, before, you'd have something like:
for (Enumeration e=container.elements();e.hasMoreElements();) {
Token t = (Token) e.nextElement();
blah blah
}
you now have:
for (Token t : container) {
blah blah
}
As best I can determine, my modified codebase is completely functionally
equivalent with the existing tool. Of course, since most of the change
is going from typeless to generically typed containers, this represents
no real functional change, since with type erasure, it produces
basically the same bytecode, except that I did replace Vector by
ArrayList and Hashtable by HashMap.
That contains all the refactored source code and a binary build of
javacc.jar.
I would encourage anybody who is using javacc in a project to drop in
this javacc.jar file in there and simply verify that it works, i.e.
generates the same code as the existing tool. (I will be very surprised
if it doesn't! :-))
I started cleaning up the code because I want to do some work on it.
Basically, I would like to be able to commit this work and continue on
it. Though I did this for my own purposes, I think that anybody who
wants to work on javacc would much prefer using this code as a starting
point.
Anyway, if this project is so dead that nobody answers me on this, I
guess I will just fork it off, probably on sourceforge, and continue
working on it there.
That said, I think that not forking is preferable.
Regards,
JR
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...
Re: Re: Contribution and question about getting involved in the project.
My time and energy are currently directed away from specification
recognition at the moment so I doubt I'll be using the cleaned-up
version any time soon, though I personally appreciate your effort. I
did verify on some small personal project, with no apparent ill effect.
The JavaCC users list is not "busy", per se, but it is more active
than many other projects I've seen. I suspect that there hasn't been
much call for new features or modifications because JavaCC is a
stable, well vetted tool with a relatively clean "user" interface
(namely, the JavaCC language and command-line utility). That said, I
do believe refactoring the "developer" interface is worthwhile, if
even simply to facilitate individual efforts to (more strongly)
integrate JavaCC with the various IDEs.
As with any project, when making code contributions you should follow
best practices by making them incrementally, ensuring they pass any
existing regression tests, and providing additional tests that
demonstrate behavioral equivalence. I recommend against a JavaCC fork
for a variety of pragmatic reasons.
Sreeni and others on the list may have further advice.
Nathan
On 3 May, 2008, at 11.44, Jonathan Revusky wrote:
> Jonathan Revusky wrote:
>> Hi,
>> I'm not 100% sure that this is the right forum, but I don't see a
>> javacc.devel list, just this user list. (There are the automated
>> notification lists for CVS and the bugtracker issues, but I can't
>> really see starting a discussion on those...)
>
> Hi again,
>
> Okay, I wrote a note in which I expressed interest in doing some
> work in the project and nobody has answered in 4 days. To be fair,
> many people have a long weekend starting on May 1, so maybe this is
> an atypical 4 days. Nonetheless, it does not give me the feeling
> that the project is very active. I was googling around for a roadmap
> or wishlist and the most recent thing I found was a thread from
> 2006, and then, AFAICS, there was no follow-through afterwards.
>
> Well, I have been doing some work on it over the last couple of
> days. What I have done mostly is I have refactored the codebase to
> use generic containers. When I first imported the codebase from CVS
> into my eclipse workspace, the project had over 800 warnings, mostly
> related to type safety. I've now got that down to 39. And actually,
> the majority of those warnings are from code that is itself
> _generated_ by JavaCC, so there was no way of getting rid of those.
>
> The lint tool in javac, i.e. compiling all the code with javac -
> Xlint, now gives 22 warnings, again, mostly from code that is itself
> javacc-generated. (Before, it stopped after 100 warnings.)
>
> A lot more cleanup and refactoring is possible, but already the code
> is significantly more readable. Where before, you had something like:
>
> Vector removeList = new Vector();
>
> now you have:
>
> List<List<RegExprSpec>> removeList = new
> ArrayList<List<RegExprSpec>>()
>
> That is actually more verbose, but it is certainly much less opaque.
> One sees the data structures that are being used. But elsewhere, the
> code becomes much less cluttered as a consequence. Once I generified
> the code, most of the casts that were all over the place became
> superfluous and could be removed. Also, I largely replaced looping
> constructs with the extended for loop in Java 1.5.
>
> So, where, before, you'd have something like:
>
> for (Enumeration e=container.elements();e.hasMoreElements();) {
> Token t = (Token) e.nextElement();
> blah blah
> }
>
> you now have:
>
> for (Token t : container) {
> blah blah
> }
>
> As best I can determine, my modified codebase is completely
> functionally equivalent with the existing tool. Of course, since
> most of the change is going from typeless to generically typed
> containers, this represents no real functional change, since with
> type erasure, it produces basically the same bytecode, except that I
> did replace Vector by ArrayList and Hashtable by HashMap.
>
> I put up an archive on a server here: http://freemarker.sourceforge.net/javacc/javacc-mod.tar.gz >
> That contains all the refactored source code and a binary build of
> javacc.jar.
>
> I would encourage anybody who is using javacc in a project to drop
> in this javacc.jar file in there and simply verify that it works,
> i.e. generates the same code as the existing tool. (I will be very
> surprised if it doesn't! :-))
>
> I started cleaning up the code because I want to do some work on it.
> Basically, I would like to be able to commit this work and continue
> on it. Though I did this for my own purposes, I think that anybody
> who wants to work on javacc would much prefer using this code as a
> starting point.
>
> Anyway, if this project is so dead that nobody answers me on this, I
> guess I will just fork it off, probably on sourceforge, and continue
> working on it there.
>
> That said, I think that not forking is preferable.
>
> Regards,
>
> JR
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@... > For additional commands, e-mail: users-help@... >
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...
Re: Re: Contribution and question about getting involved in the project.
> Jonathan Revusky wrote:
>> Hi,
>> I'm not 100% sure that this is the right forum, but I don't see a
>> javacc.devel list, just this user list. (There are the automated
>> notification lists for CVS and the bugtracker issues, but I can't
>> really see starting a discussion on those...)
>
> Hi again,
>
> Okay, I wrote a note in which I expressed interest in doing some
> work in the project and nobody has answered in 4 days. To be fair,
> many people have a long weekend starting on May 1, so maybe this is
> an atypical 4 days. Nonetheless, it does not give me the feeling
> that the project is very active. I was googling around for a roadmap
> or wishlist and the most recent thing I found was a thread from
> 2006, and then, AFAICS, there was no follow-through afterwards.
I am a "secondary" developer for JavaCC, in that I have the developer
role on java.net and have committed some cleanups, both to the code,
and to the generate code. There are a lot of changes in the current
CVS code base as compared to the last release, and (unfortunately) the
current online documentation actually refers to the CVS version of the
code. I personally think that a new release should be made soon,
however I cannot speak for the project administrators ;-)
From reading your e-mail, it looks to me like you have spent some
time to make the code Java 1.5 ready, unfortunately, at least for the
upcoming release, JavaCC is restricted to using Java 1.4 (https://javacc.dev.java.net/issues/show_bug.cgi?id=192 )
The code generated by JavaCC should be 1.5 ready if you set
JDK_VERSION=1.5
Should that still have any issues, then I will happily look at any
patch to resolve that in CVS. I will also look at removing the use of
the old Vector and Enumeration classes and replacing them with the
"modern" equivalents.
Cheers,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...
Re: Re: Contribution and question about getting involved in the project.
Jonathan Revusky wrote:
> Jonathan Revusky wrote:
>> Hi,
>>
>> I'm not 100% sure that this is the right forum, but I don't see a
>> javacc.devel list, just this user list. (There are the automated
>> notification lists for CVS and the bugtracker issues, but I can't
>> really see starting a discussion on those...)
Unfortunately the developers' list isn't visible unless you are logged
in. That does seem a bit counter-intuitive to me. The dev list would
probably be a better place for this discussion (I have copied this reply
to it).
>>
>
> Hi again,
>
> Okay, I wrote a note in which I expressed interest in doing some work in
> the project and nobody has answered in 4 days. To be fair, many people
> have a long weekend starting on May 1, so maybe this is an atypical 4
> days. Nonetheless, it does not give me the feeling that the project is
> very active. I was googling around for a roadmap or wishlist and the
> most recent thing I found was a thread from 2006, and then, AFAICS,
> there was no follow-through afterwards.
Well, there is often a flurry of development activity followed by long
periods of dormancy. But since the dev list is not public, it is
probably difficult to gauge this from the outside. I think it is rather
a pity that we haven't released anything for over two years - that does
make the project look in rather poor health.
The nearest thing we have to a roadmap is a message From the project
owner, Sreeni:
What I ABSOLUTELY don't want changed:
* ant based build. You can add maven build, but I
should still be able to just do 'ant' and get a build.
* I want to keep it a single jar
* all the generated code is stand-alone - no jars needed
Things I would be motivated to do at some point:
* generating more 1.5 code - using enums,
may be generic types etc.
* streamline EOF handling
* cleanup lexical states stuff that could be
confusing (like defining a token in multiple states)
* see if we can make the generated code table-driven
Things that interested people can fix:
* cleaning the code generator to get rid of silly
warnings - easy enough for someone to do (except
me :-)). Another possibility is to generate
some kind of annotation for all
non-user code in the grammar so that tools don't
scan that.
* TokenMgrError
* throwing IOException fom the stream classes.
* redesign/rewrite the tree builder.
* jsr45 debugging
> Well, I have been doing some work on it over the last couple of days.
> What I have done mostly is I have refactored the codebase to use generic
> containers. When I first imported the codebase from CVS into my eclipse
> workspace, the project had over 800 warnings, mostly related to type
> safety. I've now got that down to 39. And actually, the majority of
> those warnings are from code that is itself _generated_ by JavaCC, so
> there was no way of getting rid of those.
I know that TimP has been working hard to remove many of those compiler
warnings from generated code (and also from the JavaCC code base).
> The lint tool in javac, i.e. compiling all the code with javac -Xlint,
> now gives 22 warnings, again, mostly from code that is itself
> javacc-generated. (Before, it stopped after 100 warnings.)
>
> A lot more cleanup and refactoring is possible, but already the code is
> significantly more readable. Where before, you had something like:
>
> Vector removeList = new Vector();
>
> now you have:
>
> List<List<RegExprSpec>> removeList = new ArrayList<List<RegExprSpec>>()
The JavaCC codebase could certainly do with some cleanup and
"modernisation".
As far as I understand it, the plans for JavaCC with respect to the
Java 1.5 constructs are:
* JavaCC 4.1 will generate Java 1.4 code by default, but can
optionally generate "modern" Java 1.5 code (command-line or grammar option).
> I would encourage anybody who is using javacc in a project to drop in
> this javacc.jar file in there and simply verify that it works, i.e.
> generates the same code as the existing tool. (I will be very surprised
> if it doesn't! :-))
I'll try to build the Debian Java libraries using your new jar - that is
usually a good test. Did you intentionally remove build.xml from the source?
> I started cleaning up the code because I want to do some work on it.
> Basically, I would like to be able to commit this work and continue on
> it. Though I did this for my own purposes, I think that anybody who
> wants to work on javacc would much prefer using this code as a starting
> point.
>
> Anyway, if this project is so dead that nobody answers me on this, I
> guess I will just fork it off, probably on sourceforge, and continue
> working on it there.
>
> That said, I think that not forking is preferable.
>
> Regards,
>
> JR
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...
Re: Contribution and question about getting involved in the project.
> Hi Jonathon,
>
> On 3 May 2008, at 19:44, Jonathan Revusky wrote:
>
>> Jonathan Revusky wrote:
>>> Hi,
>>> I'm not 100% sure that this is the right forum, but I don't see a
>>> javacc.devel list, just this user list. (There are the automated
>>> notification lists for CVS and the bugtracker issues, but I can't
>>> really see starting a discussion on those...)
>>
>> Hi again,
>>
>> Okay, I wrote a note in which I expressed interest in doing some work
>> in the project and nobody has answered in 4 days. To be fair, many
>> people have a long weekend starting on May 1, so maybe this is an
>> atypical 4 days. Nonetheless, it does not give me the feeling that the
>> project is very active. I was googling around for a roadmap or
>> wishlist and the most recent thing I found was a thread from 2006, and
>> then, AFAICS, there was no follow-through afterwards.
>
> I am a "secondary" developer for JavaCC, in that I have the developer
> role on java.net and have committed some cleanups, both to the code, and
> to the generate code. There are a lot of changes in the current CVS code
> base as compared to the last release, and (unfortunately) the current
> online documentation actually refers to the CVS version of the code. I
> personally think that a new release should be made soon, however I
> cannot speak for the project administrators ;-)
>
> From reading your e-mail, it looks to me like you have spent some time
> to make the code Java 1.5 ready, unfortunately, at least for the
> upcoming release, JavaCC is restricted to using Java 1.4
> (https://javacc.dev.java.net/issues/show_bug.cgi?id=192)
Let me see, if I understand you right, you are saying that the code of
the tool itself must, for now, be 1.4 compatible.
Of course, maybe I am misunderstanding something here. It makes some
sense for the code that JavaCC *generates* to be compilable in JDK 1.4
or maybe even lower. OTOH, the tool itself running on older JVM's is not
something that has any value whatsoever AFAICS. JavaCC is a tool used by
java developers. ALL java developers have a 1.5 JVM they can use. It's
simply not an issue.
Moreover, if it were an issue, one could simply use a tool like
retrotranslator and produce a .jar file that would work under the older
JVM's. In my experience the tool works quite well.
Anyway, note that the code I provided *generates* the same output as the
existing tool, which does not require 1.5. It's the generated code
requiring 1.5 or not that would be an issue.
Now, maybe I'm missing some point about this. I would scan the archives
of the dev list to see if this was really debated, but the dev list
apparently exists but is private and I can't look at the archives.
> The code generated by JavaCC should be 1.5 ready if you set JDK_VERSION=1.5
>
> Should that still have any issues, then I will happily look at any patch
> to resolve that in CVS. I will also look at removing the use of the old
> Vector and Enumeration classes and replacing them with the "modern"
> equivalents.
Well, I have no specific issue with JavaCC as a user. The question I
posed specifically was about getting involved in the project. I have
taken pains not to show up empty handed. In my first message posted
here, I provided a template-based implementation of
org.javacc.jjdoc.Generator so that JJDoc can generate output based on an
editable freemarker template. Now, I have provided a refactoring of all
the code to use 1.5 generic containers.
I am quite confident that both of the above are unambiguous
improvements. If this project is in a state where somebody can come in
and simply give you this and you don't take advantage of it, then,
frankly, it's hopeless, and I'll just fork off somewhere and work on it
there. Now, I hope that doesn't sound too aggressive or whatever, but I
think it's reasonable just to say where things stand.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...
Re: Re: Contribution and question about getting involved in the project.
Some parts of this message have been removed.
Learn more about Nabble's 
TemplateBasedJJDoc.zip (5K) 