Problem with file paths when deploying to virtual host

[codger4110]

Hi, all.

I'm having a lot of trouble with setting up ProgramD on a shared host.  I have a shared Red Hat server where my home directory is /home/r/rascal.  I'm trying to deploy to Tomcat 5.0.25 with JDK 1.5.

The problem is that the default directories in CoreSettings.java, core.xml, and the other files are all written to assume you have access to the root directory and everything below it.  So I've had to do a lot of rearranging since, for example, /var/programd/ffm does not exist and is not accessible to me.  I've had to modify build.xml to put things like the conf directory and aiml directories into the war file, so they get deployed to my web host, and then point everything to them properly, so that it's all consistent with the new layout.

So, my first question is, have I missed something?  This seems like an awful lot of work, and I'm just wondering if there's some easier way to get ProgramD running on a web host where you don't own everything from the root directory down.  Why is ProgramD written in such a way that conf, aiml, and var are decoupled from the rest of the deployment?

Now, I've made a lot of progress, but I've gotten stuck.  The problem comes during start-up and relates to creating the SampleBot directory for the FlatFileMultiplexor.  The issue is that the URL of the web host seems to be getting turned into a file path in such a way that I'm once again stuck because I don't own the entire filesystem from / on down.  The stacktrace is this:


org.aitools.programd.util.UserError: Could not create FlatFileMultiplexor predicates file directory "/www.private.rascal.com/programd/WEB-INF/var/ffm/SampleBot/89B22BB43D819CF05198FA0638B6E0A8.predicates".
        org.aitools.programd.util.FileManager.checkOrCreate(FileManager.java:281)
        org.aitools.programd.multiplexor.FlatFileMultiplexor.loadPredicates(FlatFileMultiplexor.java:176)
        org.aitools.programd.multiplexor.FlatFileMultiplexor.loadPredicate(FlatFileMultiplexor.java:150)
        org.aitools.programd.multiplexor.PredicateMaster.get(PredicateMaster.java:216)
        org.aitools.programd.server.tags.Get.doTag(Get.java:49)
        org.apache.jsp.pages.TalkToBot_jspx._jspx_meth_aiml_get_0(TalkToBot_jspx.java:253)
        org.apache.jsp.pages.TalkToBot_jspx._jspService(TalkToBot_jspx.java:102)
        org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:298)
        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
        org.aitools.programd.server.servlet.TalkToBotServlet.forward(TalkToBotServlet.java:203)
        org.aitools.programd.server.servlet.TalkToBotServlet.setupBot(TalkToBotServlet.java:198)
        org.aitools.programd.server.servlet.TalkToBotServlet.doGet(TalkToBotServlet.java:78)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:810)


As you can see, I've managed to relocate the /var/ffm/SampleBot directory to within my personal filespace, but the URL of my domain has made its way into the file path, and I don't know how to get it out of there.  If this were a relative path, I'd be in good shape, but somehow it's added the "/www.private.rascal.com" part from the URL to the beginning of the path, so it's absolute.  I've traced this back to FileManager, FlatFileMultiplexor, and URLTools, but I just don't see the fix.  The problem already exists when it gets to the FlatFileMultiplexor constructor.  In the constructor, I'm printing out this.ffmDirName, and it's set to "/www.private.rascal.com/programd/WEB-INF/var/ffm".  I don't see what to do to make this relative, other than some risky repairs on ffmDirName.  Is there a way to get this value set properly in the first place, so that it's relative to the current Tomcat directory, or something like that?  It seems like that was successfully done to locate core.!
 xml, so why not here?

Any help (and advice on what the easy way to do this would have been ;) would be greatly appreciated!  Sorry for being too wordy.  I'm just trying to be clear.

Cheers,
Rascal


_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[David Wallace Croft-2]

--- codger4110@... wrote:
> I'm having a lot of trouble with setting up ProgramD on a shared
> host.  I have a shared Red Hat server where my home directory is
> /home/r/rascal.  I'm trying to deploy to Tomcat 5.0.25 with JDK 1.5.



After spending many frustrating hours trying to get a Java CMS to work
with shared hosting, I gave up and looked for something else.  Here is
what I found:
http://eapps.com/ManagedHosting/VirtualPrivateServer.jsp
http://eapps.com/Docs/VPSStandardPrices.jsp

--
David Wallace Croft / [phone] m
http://www.CroftSoft.com/people/david/
_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[codger4110]

I'd really rather get the software working than go through changing hosts.  :-)

Thanks, anyway!


-----Original Message-----

>From: David Wallace Croft <david_wallace_croft@...>
>Sent: Nov 14, 2006 5:55 PM
>To: Program D <programd@...>
>Subject: Re: [programd] Problem with file paths when deploying to virtual host
>
>--- codger4110@... wrote:
>> I'm having a lot of trouble with setting up ProgramD on a shared
>> host.  I have a shared Red Hat server where my home directory is
>> /home/r/rascal.  I'm trying to deploy to Tomcat 5.0.25 with JDK 1.5.
>
>
>
>After spending many frustrating hours trying to get a Java CMS to work
>with shared hosting, I gave up and looked for something else.  Here is
>what I found:
>http://eapps.com/ManagedHosting/VirtualPrivateServer.jsp
>http://eapps.com/Docs/VPSStandardPrices.jsp
>
>--
>David Wallace Croft / [phone] m
>http://www.CroftSoft.com/people/david/
>_______________________________________________
>programd mailing list
>programd@...
>http://aitools.org/mailman/listinfo/programd

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[Noel Bush]

Rascal,

Paths are the bane of my existence, and I seem to be continually
reworking how they get handled, to deal with more and more subtleties.
I also don't really understand the most desirable way to package the
app.  I guess my idea was that people would want a .war file they could
just deploy without having to rebuild everytime they change AIML files
or other configuration.  As I've been told, the proper way to handle all
of this is for everything to be in a database.

But I have done a fair amount of twiddling since the 4.6 release, and if
you are adventurous enough to try out CVS HEAD, you may get better
results with some of the problems you've mentioned.

I have had nice experiences with eApps, by the way.  But I understand
your desire not to change hosts.

Please let me know if the stuff in cvs works any better for you, and/or
if you can suggest a better way of arranging things.  I really don't
know what the norm is out there for deployment of Program D as a web
app, or if there even is a norm.  I know of several different
deployments that have taken quite different approaches, all depending on
the type of environment/enterprise that they're in with.

Noel

codger4110@... wrote:

> I'd really rather get the software working than go through changing hosts.  :-)
>
> Thanks, anyway!
>
>
> -----Original Message-----
>> From: David Wallace Croft <david_wallace_croft@...>
>> Sent: Nov 14, 2006 5:55 PM
>> To: Program D <programd@...>
>> Subject: Re: [programd] Problem with file paths when deploying to virtual host
>>
>> --- codger4110@... wrote:
>>> I'm having a lot of trouble with setting up ProgramD on a shared
>>> host.  I have a shared Red Hat server where my home directory is
>>> /home/r/rascal.  I'm trying to deploy to Tomcat 5.0.25 with JDK 1.5.
>>
>>
>> After spending many frustrating hours trying to get a Java CMS to work
>> with shared hosting, I gave up and looked for something else.  Here is
>> what I found:
>> http://eapps.com/ManagedHosting/VirtualPrivateServer.jsp
>> http://eapps.com/Docs/VPSStandardPrices.jsp
>>
>> --
>> David Wallace Croft / [phone] m
>> http://www.CroftSoft.com/people/david/
>> _______________________________________________
>> programd mailing list
>> programd@...
>> http://aitools.org/mailman/listinfo/programd
>
>
>
> _______________________________________________
> programd mailing list
> programd@...
> http://aitools.org/mailman/listinfo/programd

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[codger4110]

Noel,

Thanks for your reply.  You've done a heck of a job, so don't let paths get you down.  Everybody's got to have one weakness.  :-)

In general, I would like to be able to deploy a complete, working app to any server with just a war file.  If I want to just deploy new AIML files, I would like another ant target that will deploy just those, without having to reinstall the entire war.  I like the fact that it's not all dependent on a database, so I would encourage you to avoid that dependency.

The thing that I'm battling now has to do with confusion over absolute vs. relative paths.  And, somehow, the URL domain of my web app is making its way into the file path for the SampleBot predicates.  That's not right, but I'm not sure how it happens.  I'm reluctant to start playing with edge releases, as I'm trying to accomplish something quickly.  I'm tempted to just hack the properties after the fact to fix them, but it would be nice to understand how the program picks up the web app URL and how it decides whether or not a path is relative vs. absolute.  I think if I could manage that part of the initialization, I could configure the file paths to go where I need them to.

So, question 1 is where is it picking up the "/www.private.rascal.com" part of the URL and adding it to the file directory (actually to ffmDirName)?  I never explicitly set that, so it's retrieving it from somewhere.

And question 2 is why does it think that "WEB-INF/var/ffm" needs to be made absolute, rather than just leaving it as a relative path?  It seems like other paths, such as the path to core.xml, were left as relative to the Tomcat app directory, so why is ffmDirName treated differently?

I think if I knew the answer to those two questions, I might be able to get past this.

Many thanks,
Rascal


-----Original Message-----

>From: Noel Bush <noel@...>
>Sent: Nov 14, 2006 8:53 PM
>To: Program D <programd@...>
>Subject: Re: [programd] Problem with file paths when deploying to virtual host
>
>Rascal,
>
>Paths are the bane of my existence, and I seem to be continually
>reworking how they get handled, to deal with more and more subtleties.
>I also don't really understand the most desirable way to package the
>app.  I guess my idea was that people would want a .war file they could
>just deploy without having to rebuild everytime they change AIML files
>or other configuration.  As I've been told, the proper way to handle all
>of this is for everything to be in a database.
>
>But I have done a fair amount of twiddling since the 4.6 release, and if
>you are adventurous enough to try out CVS HEAD, you may get better
>results with some of the problems you've mentioned.
>
>I have had nice experiences with eApps, by the way.  But I understand
>your desire not to change hosts.
>
>Please let me know if the stuff in cvs works any better for you, and/or
>if you can suggest a better way of arranging things.  I really don't
>know what the norm is out there for deployment of Program D as a web
>app, or if there even is a norm.  I know of several different
>deployments that have taken quite different approaches, all depending on
>the type of environment/enterprise that they're in with.
>
>Noel
>
>codger4110@... wrote:
>> I'd really rather get the software working than go through changing hosts.  :-)
>>
>> Thanks, anyway!
>>
>>
>> -----Original Message-----
>>> From: David Wallace Croft <david_wallace_croft@...>
>>> Sent: Nov 14, 2006 5:55 PM
>>> To: Program D <programd@...>
>>> Subject: Re: [programd] Problem with file paths when deploying to virtual host
>>>
>>> --- codger4110@... wrote:
>>>> I'm having a lot of trouble with setting up ProgramD on a shared
>>>> host.  I have a shared Red Hat server where my home directory is
>>>> /home/r/rascal.  I'm trying to deploy to Tomcat 5.0.25 with JDK 1.5.
>>>
>>>
>>> After spending many frustrating hours trying to get a Java CMS to work
>>> with shared hosting, I gave up and looked for something else.  Here is
>>> what I found:
>>> http://eapps.com/ManagedHosting/VirtualPrivateServer.jsp
>>> http://eapps.com/Docs/VPSStandardPrices.jsp
>>>
>>> --
>>> David Wallace Croft / [phone] m
>>> http://www.CroftSoft.com/people/david/
>>> _______________________________________________
>>> programd mailing list
>>> programd@...
>>> http://aitools.org/mailman/listinfo/programd
>>
>>
>>
>> _______________________________________________
>> programd mailing list
>> programd@...
>> http://aitools.org/mailman/listinfo/programd
>_______________________________________________
>programd mailing list
>programd@...
>http://aitools.org/mailman/listinfo/programd

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[mehri]

<quote>
In general, I would like to be able to deploy a complete, working app
to any server with just a war file.  If I want to just deploy new AIML
files, I would like another ant target that will deploy just those,
without having to reinstall the entire war.  I like the fact that it's
not all dependent on a database, so I would encourage you to avoid that
dependency.
</quote>

My humble opinion:

Deploying a complete working app with just a war file is just the first step.  The second step is always configuration after installation.  After that, the user always wants to reconfigure time after time.  Again and again.  Most J2EE programmers usually write web page(s) for the administrator to be able to conduct configuration.  Steps typically done I've seen:

1) The user downloads the .war file non-exploded/exploded

2) Deploys it on their app server.
2a) When the app server is being deployed it picks up a few environment variables along the way.  Environment variables such as the current user's name and their home directory.  It finds a convient spot (unix it can be "${HOME}/.<conf-directory>", under Windows it can be C:\Documents and Settings\${USER}\Application Data\${YOUR_APPLICATION}) and stores writable configuration files.  These will be log4j configuration files, AIML files, etc...

3) After deployment of the .war file the user opens up a browser on any client machine to the configuration web page URL.  The .war file on the backend server sees that this is the first time run for deployment.  It prompts for admin user id/password.  User puts these values in and they are stored encrypted in the convient spot mentioned in 2a.

4) Next configuration screen is the *optional* database connection info.  This is completely optional.  It gives the user the option to create either new DB tables for configuration info or to read pre-existing info from the DB.  If the user says, "yea, I have a DB.  Here's my connection info, password, user id, IP, port, etc..." then the war opens up an initial connection reads the DB and scans it for properties, configuration, etc... If the user says, "yea, I have a DB, create everything new" it'll give a few more web page prompts with defaults for DB table names, schema, etc..."  then create those tables.

5) The DB part is optional.  If the user selects "no, I don't need a DB.  I want flat files".  It then brings up configuration screens for where the configuration files are to be stored or for the user to point to pre-existing configuration files they already have.

6) Regardless of whether a DB or flat file is choosen the war file can't modify its own files in its war.  It insteads writes the bare minimum of either DB connection info or paths to configuration files in the spot it found for its self in 2a).  This way if a new war version'ed file is released it can just be dropped into the app server and redeploy.  The upgraded version will reread those properties and rehook up.  Upgrading is easier at this point.

7) The next time the user goes to the configuration screen it'll have the values put in previously (It either saved them out in the DB or it saved them to the flat files from the first run.  The user can go through and change the configuration as they always do no problems.  It'll re-save either to the flat files or to the DB.  One note is that for the configuration screen usually there's a "lock and edit" added.  This is to allow only one client machine at a time to update configuration information so that if two people from two different computers try to make changes they cannot stomp on each other.  Also, a log of what adminstrations had taken place with date stamps is usually convient.  That way the user can see someone logged in yesterday and changed the bot properties file.

----

Finally....A few notes on using flat files versus a DB.

If you have just one machine you're running as a bot server flat files are fine for configuration.  When the bot is running it'll store out user predicates per user id in the flat files.  It'll store its bot predicates in a flat file.  These will also remain in the bot's memory.  If you shut down your server and restart it, it'll reload these same flat files into memory and everything will start up fine.  I won't get into stateful beans at this point but could use them :-)

If you begin putting in multiple machines to handle redudancy, load balancing, a DB is your best bet.  Either that or you store the flat files on a shared drive to another machine (be warned about the shared drive on a machine though, because you have a single point of failure which defeats the purpose of redudancy doesn't it?)

Enterprise usually means that you are using a DB and multiple machines for redudancy and load balancing.  What usually happens is you have a forest of machines you start up.  They've all been through the initial configuration web pages to give them the bare bones info of where the main DB is located. Upon startup they read through the DB and load user properties, bot properties etc...Some of them might even know that they'll only serve users xyz and load only those user's pervious predicates.

One admin machine that redirects to the forest is started.  It's job is to redirect user's to each individual machine and it has the smarts for load balancing and redundancy.  It doesn't really do any bot work.  It's only job is redirection to keep so it has the least chance of dying on you.  If one of the machine's in the forest goes down it redirects to another.

The DB its self (depending on the type of DB you have) usually handles its own redundancy and load balancing.  So admin that seperate and let that handle its self.  In the background for cron jobs you have the DB's data dumped and archived, backed up, etc... at night.

Note, that with all these machines running and you go to an admin screen (any screen at this point of any of the machines in the forest) you can change a bot property in the DB and know that all the machines will see this change.  No worries about having to change *all* configuration files on all your forest machines.  

Also, if you need to do an upgrade, you can selectively shut down one machine in the forest at a time redeploy the new war file restart the machine and move on to the next machine.  Zero downtime.

Also, when the user predicates are stored they're stored out in the DB.  If that machine goes down another can take it's place and load that user's predicates from the DB.  No need to worry about syncing up that info between the machines.  Lastly, you have the power of SQL to conduct queries on what's being said, who's saying it, what's going on with your admin, etc...  You can always wrap that up nicely into a nice web application so anywhere else you can check health of your forest what user's are connected, etc...


Just my 2 cents for what it's worth.

----- Original Message ----
From: "codger4110@..." <codger4110@...>
To: Program D <programd@...>; noel@...
Sent: Wednesday, November 15, 2006 12:40:14 AM
Subject: Re: [programd] Problem with file paths when deploying to virtual host

Noel,

Thanks for your reply.  You've done a heck of a job, so don't let paths get you down.  Everybody's got to have one weakness.  :-)

In general, I would like to be able to deploy a complete, working app to any server with just a war file.  If I want to just deploy new AIML files, I would like another ant target that will deploy just those, without having to reinstall the entire war.  I like the fact that it's not all dependent on a database, so I would encourage you to avoid that dependency.

The thing that I'm battling now has to do with confusion over absolute vs. relative paths.  And, somehow, the URL domain of my web app is making its way into the file path for the SampleBot predicates.  That's not right, but I'm not sure how it happens.  I'm reluctant to start playing with edge releases, as I'm trying to accomplish something quickly.  I'm tempted to just hack the properties after the fact to fix them, but it would be nice to understand how the program picks up the web app URL and how it decides whether or not a path is relative vs. absolute.  I think if I could manage that part of the initialization, I could configure the file paths to go where I need them to.

So, question 1 is where is it picking up the "/www.private.rascal.com" part of the URL and adding it to the file directory (actually to ffmDirName)?  I never explicitly set that, so it's retrieving it from somewhere.

And question 2 is why does it think that "WEB-INF/var/ffm" needs to be made absolute, rather than just leaving it as a relative path?  It seems like other paths, such as the path to core.xml, were left as relative to the Tomcat app directory, so why is ffmDirName treated differently?

I think if I knew the answer to those two questions, I might be able to get past this.

Many thanks,
Rascal


-----Original Message-----

>From: Noel Bush <noel@...>
>Sent: Nov 14, 2006 8:53 PM
>To: Program D <programd@...>
>Subject: Re: [programd] Problem with file paths when deploying to virtual host
>
>Rascal,
>
>Paths are the bane of my existence, and I seem to be continually
>reworking how they get handled, to deal with more and more subtleties.
>I also don't really understand the most desirable way to package the
>app.  I guess my idea was that people would want a .war file they could
>just deploy without having to rebuild everytime they change AIML files
>or other configuration.  As I've been told, the proper way to handle all
>of this is for everything to be in a database.
>
>But I have done a fair amount of twiddling since the 4.6 release, and if
>you are adventurous enough to try out CVS HEAD, you may get better
>results with some of the problems you've mentioned.
>
>I have had nice experiences with eApps, by the way.  But I understand
>your desire not to change hosts.
>
>Please let me know if the stuff in cvs works any better for you, and/or
>if you can suggest a better way of arranging things.  I really don't
>know what the norm is out there for deployment of Program D as a web
>app, or if there even is a norm.  I know of several different
>deployments that have taken quite different approaches, all depending on
>the type of environment/enterprise that they're in with.
>
>Noel
>
>codger4110@... wrote:
>> I'd really rather get the software working than go through changing hosts.  :-)
>>
>> Thanks, anyway!
>>
>>
>> -----Original Message-----
>>> From: David Wallace Croft <david_wallace_croft@...>
>>> Sent: Nov 14, 2006 5:55 PM
>>> To: Program D <programd@...>
>>> Subject: Re: [programd] Problem with file paths when deploying to virtual host
>>>
>>> --- codger4110@... wrote:
>>>> I'm having a lot of trouble with setting up ProgramD on a shared
>>>> host.  I have a shared Red Hat server where my home directory is
>>>> /home/r/rascal.  I'm trying to deploy to Tomcat 5.0.25 with JDK 1.5.
>>>
>>>
>>> After spending many frustrating hours trying to get a Java CMS to work
>>> with shared hosting, I gave up and looked for something else.  Here is
>>> what I found:
>>> http://eapps.com/ManagedHosting/VirtualPrivateServer.jsp
>>> http://eapps.com/Docs/VPSStandardPrices.jsp
>>>
>>> --
>>> David Wallace Croft / [phone] m
>>> http://www.CroftSoft.com/people/david/
>>> _______________________________________________
>>> programd mailing list
>>> programd@...
>>> http://aitools.org/mailman/listinfo/programd
>>
>>
>>
>> _______________________________________________
>> programd mailing list
>> programd@...
>> http://aitools.org/mailman/listinfo/programd
>_______________________________________________
>programd mailing list
>programd@...
>http://aitools.org/mailman/listinfo/programd

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd





 
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[David Wallace Croft-2]

I read in the documentation that predicates can be stored in a database
and I found some of the Java/SQL code for that.  Is the plan to permit
storing properties in the database sometime in the future too?  How
about categories?


--
David Wallace Croft / [phone] m
http://www.CroftSoft.com/people/david/
_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd