Locale and Date Optimizations

[Justin Plock]

Hi Everyone,
   I was wondering if anyone was working on or knew of a way to optimize
the Zend_Date, Zend_Locale and Zend_Translate class. Currently, I've
built a wrapper class around Zend_Cache, that after it successfully
connects to Memcached, it execute the following:

try {
     /**
      * @see Zend_Cache
      */
     require 'Zend/Cache.php';

     $this->_cache = Zend_Cache::factory('Core', $config['type'],
$frontend, $backend);
     if ($frontend['caching'] && $this->_cache->save('test', 'test')) {
         /**
          * @see Zend_Db_Table_Abstract
          */
         require 'Zend/Db/Table/Abstract.php';
         Zend_Db_Table_Abstract::setDefaultMetadataCache($this->_cache);

         /**
          * @see Zend_Locale
          */
         require 'Zend/Locale.php';
         Zend_Locale::setCache($this->_cache);

         /**
          * @see Zend_Translate
          */
         require 'Zend/Translate.php';
         Zend_Translate::setCache($this->_cache);

         /**
          * @see Zend_Date
          */
         require 'Zend/Date.php';
         Zend_Date::setOptions(array('cache'=>$this->_cache));
     }
     else {
         $this->_cache = null;
     }
}
catch (Exception $e) {
}

I've been benchmarking and optimizing my app using xdebug and
kcachegrind and this series of require commands (to utilize APC opcode
caching and avoid Zend_Loader) and those caching calls consumes up a
large portion of each of my requests. Zend_Db_Table_Abstract isn't too
big of a hit considering I'm using Zend_Db everywhere already. Is there
anyway we can, for example, on the Zend_Translate_Adapter_Gettext class,
to use the gettext() extension if it's built-in? I'm suspecting most of
the performance hit is from loading the Locale data from the XML files.
Memcached will take over after the initial request, but there are a lot
of require_once calls occurring throughout these classes.

I'm curious what others are up to and have seen.

Thanks.

-Justin

[thomasW]

Hy Justin,

First:
No, the build in gettext extension won't work... it's not really locale
aware and has known problem in multithreaded servers like apache.
And what do you expect to be faster when using build in gettext as adapter ?
Gettext is the fastest available adapter.

Your files are only read once if you use caching but also only if you use
directory search.
I can't follow your argument that you have a performance loss in every
request when you use cache.
Silly question... are you sure your cache works ?
Maybe you are adding new translation files after initiating Zend_Translate ?

Second:
You've made performance testing and "expect" xml files to be the reason for
your performance loss ?
Have you tested it, or are you expecting ? You conflict yourself on this
one...
The XML files are not read when they are not used. Also what xml file is
read depends on your locale. We do not read all 130 xml files if you expect
us to do so. :-)

Also already read values are kept in cache and for this data the file is not
opened anymore.
Within here it seems that you eighter have not used cached data, or your
cache is too short as requested data are only read once.

Greetings
Thomas Weidner, I18N Team Leader
http://www.thomasweidner.com

----- Original Message -----
From: "Justin Plock" <jplock@...>
To: <fw-i18n@...>
Sent: Thursday, April 10, 2008 3:01 AM
Subject: [fw-i18n] Locale and Date Optimizations

> Hi Everyone,
>   I was wondering if anyone was working on or knew of a way to optimize
> the Zend_Date, Zend_Locale and Zend_Translate class. Currently, I've built
> a wrapper class around Zend_Cache, that after it successfully connects to
> Memcached, it execute the following:
>
> try {
>     /**
>      * @see Zend_Cache
>      */
>     require 'Zend/Cache.php';
>
>     $this->_cache = Zend_Cache::factory('Core', $config['type'],
> $frontend, $backend);
>     if ($frontend['caching'] && $this->_cache->save('test', 'test')) {
>         /**
>          * @see Zend_Db_Table_Abstract
>          */
>         require 'Zend/Db/Table/Abstract.php';
>         Zend_Db_Table_Abstract::setDefaultMetadataCache($this->_cache);
>
>         /**
>          * @see Zend_Locale
>          */
>         require 'Zend/Locale.php';
>         Zend_Locale::setCache($this->_cache);
>
>         /**
>          * @see Zend_Translate
>          */
>         require 'Zend/Translate.php';
>         Zend_Translate::setCache($this->_cache);
>
>         /**
>          * @see Zend_Date
>          */
>         require 'Zend/Date.php';
>         Zend_Date::setOptions(array('cache'=>$this->_cache));
>     }
>     else {
>         $this->_cache = null;
>     }
> }
> catch (Exception $e) {
> }
>
> I've been benchmarking and optimizing my app using xdebug and kcachegrind
> and this series of require commands (to utilize APC opcode caching and
> avoid Zend_Loader) and those caching calls consumes up a large portion of
> each of my requests. Zend_Db_Table_Abstract isn't too big of a hit
> considering I'm using Zend_Db everywhere already. Is there anyway we can,
> for example, on the Zend_Translate_Adapter_Gettext class, to use the
> gettext() extension if it's built-in? I'm suspecting most of the
> performance hit is from loading the Locale data from the XML files.
> Memcached will take over after the initial request, but there are a lot of
> require_once calls occurring throughout these classes.
>
> I'm curious what others are up to and have seen.
>
> Thanks.
>
> -Justin

[Justin Plock]

Hi Thomas,
  As for the gettext issue, I'm not concerned about multithreaded
apache because I only ever use the prefork version.  My point is with
my email, is that for the entire execution time of my application
(which includes connecting to a database, a few simple queries which
are also cached with Memcached, and loading a Smarty template), the
majority of the execution time is spent setting up these caches.  You
are correct, after the files are loaded into the cache, the pages are
quicker than not being in the cache.

I was just concerned with the number of files being loaded by each of
these modules and if there was any plan to try and optimize it a bit?

Thanks.

-Justin

On Thu, Apr 10, 2008 at 4:41 AM, Thomas Weidner <thomas.weidner@...> wrote:

> Hy Justin,
>
>  First:
>  No, the build in gettext extension won't work... it's not really locale
> aware and has known problem in multithreaded servers like apache.
>  And what do you expect to be faster when using build in gettext as adapter
> ?
>  Gettext is the fastest available adapter.
>
>  Your files are only read once if you use caching but also only if you use
> directory search.
>  I can't follow your argument that you have a performance loss in every
> request when you use cache.
>  Silly question... are you sure your cache works ?
>  Maybe you are adding new translation files after initiating Zend_Translate
> ?
>
>  Second:
>  You've made performance testing and "expect" xml files to be the reason for
> your performance loss ?
>  Have you tested it, or are you expecting ? You conflict yourself on this
> one...
>  The XML files are not read when they are not used. Also what xml file is
> read depends on your locale. We do not read all 130 xml files if you expect
> us to do so. :-)
>
>  Also already read values are kept in cache and for this data the file is
> not opened anymore.
>  Within here it seems that you eighter have not used cached data, or your
> cache is too short as requested data are only read once.
>
>  Greetings
>  Thomas Weidner, I18N Team Leader
>  http://www.thomasweidner.com
>
>  ----- Original Message ----- From: "Justin Plock" <jplock@...>
>  To: <fw-i18n@...>
>  Sent: Thursday, April 10, 2008 3:01 AM
>  Subject: [fw-i18n] Locale and Date Optimizations
>
>
>
> > Hi Everyone,
> >  I was wondering if anyone was working on or knew of a way to optimize the
> Zend_Date, Zend_Locale and Zend_Translate class. Currently, I've built a
> wrapper class around Zend_Cache, that after it successfully connects to
> Memcached, it execute the following:
> >
> > try {
> >    /**
> >     * @see Zend_Cache
> >     */
> >    require 'Zend/Cache.php';
> >
> >    $this->_cache = Zend_Cache::factory('Core', $config['type'], $frontend,
> $backend);
> >    if ($frontend['caching'] && $this->_cache->save('test', 'test')) {
> >        /**
> >         * @see Zend_Db_Table_Abstract
> >         */
> >        require 'Zend/Db/Table/Abstract.php';
> >        Zend_Db_Table_Abstract::setDefaultMetadataCache($this->_cache);
> >
> >        /**
> >         * @see Zend_Locale
> >         */
> >        require 'Zend/Locale.php';
> >        Zend_Locale::setCache($this->_cache);
> >
> >        /**
> >         * @see Zend_Translate
> >         */
> >        require 'Zend/Translate.php';
> >        Zend_Translate::setCache($this->_cache);
> >
> >        /**
> >         * @see Zend_Date
> >         */
> >        require 'Zend/Date.php';
> >        Zend_Date::setOptions(array('cache'=>$this->_cache));
> >    }
> >    else {
> >        $this->_cache = null;
> >    }
> > }
> > catch (Exception $e) {
> > }
> >
> > I've been benchmarking and optimizing my app using xdebug and kcachegrind
> and this series of require commands (to utilize APC opcode caching and avoid
> Zend_Loader) and those caching calls consumes up a large portion of each of
> my requests. Zend_Db_Table_Abstract isn't too big of a hit considering I'm
> using Zend_Db everywhere already. Is there anyway we can, for example, on
> the Zend_Translate_Adapter_Gettext class, to use the gettext() extension if
> it's built-in? I'm suspecting most of the performance hit is from loading
> the Locale data from the XML files. Memcached will take over after the
> initial request, but there are a lot of require_once calls occurring
> throughout these classes.
> >
> > I'm curious what others are up to and have seen.
> >
> > Thanks.
> >
> > -Justin
> >
>
>

[thomasW]

Justin,

as you did not gave any details it's impossible to help.

Related to files and cache...
If you are, for example, loading about 100 translation files per directory
search which loads about 10.000 translations there is an emminent delay at
the first request and when the cache expires.
There are several other things which have an impact on performance in
general. So without any details there is no way.

Looking only at the number of files being loaded in not the right way to
measure performance.

Greetings
Thomas Weidner, I18N Team Leader
http://www.thomasweidner.com

----- Original Message -----
From: "Justin Plock" <jplock@...>
To: "Thomas Weidner" <thomas.weidner@...>
Cc: <fw-i18n@...>
Sent: Thursday, April 10, 2008 9:41 PM
Subject: Re: [fw-i18n] Locale and Date Optimizations

> Hi Thomas,
>  As for the gettext issue, I'm not concerned about multithreaded
> apache because I only ever use the prefork version.  My point is with
> my email, is that for the entire execution time of my application
> (which includes connecting to a database, a few simple queries which
> are also cached with Memcached, and loading a Smarty template), the
> majority of the execution time is spent setting up these caches.  You
> are correct, after the files are loaded into the cache, the pages are
> quicker than not being in the cache.
>
> I was just concerned with the number of files being loaded by each of
> these modules and if there was any plan to try and optimize it a bit?
>
> Thanks.
>
> -Justin
>
> On Thu, Apr 10, 2008 at 4:41 AM, Thomas Weidner <thomas.weidner@...>
> wrote:
>> Hy Justin,
>>
>>  First:
>>  No, the build in gettext extension won't work... it's not really locale
>> aware and has known problem in multithreaded servers like apache.
>>  And what do you expect to be faster when using build in gettext as
>> adapter
>> ?
>>  Gettext is the fastest available adapter.
>>
>>  Your files are only read once if you use caching but also only if you
>> use
>> directory search.
>>  I can't follow your argument that you have a performance loss in every
>> request when you use cache.
>>  Silly question... are you sure your cache works ?
>>  Maybe you are adding new translation files after initiating
>> Zend_Translate
>> ?
>>
>>  Second:
>>  You've made performance testing and "expect" xml files to be the reason
>> for
>> your performance loss ?
>>  Have you tested it, or are you expecting ? You conflict yourself on this
>> one...
>>  The XML files are not read when they are not used. Also what xml file is
>> read depends on your locale. We do not read all 130 xml files if you
>> expect
>> us to do so. :-)
>>
>>  Also already read values are kept in cache and for this data the file is
>> not opened anymore.
>>  Within here it seems that you eighter have not used cached data, or your
>> cache is too short as requested data are only read once.
>>
>>  Greetings
>>  Thomas Weidner, I18N Team Leader
>>  http://www.thomasweidner.com
>>
>>  ----- Original Message ----- From: "Justin Plock" <jplock@...>
>>  To: <fw-i18n@...>
>>  Sent: Thursday, April 10, 2008 3:01 AM
>>  Subject: [fw-i18n] Locale and Date Optimizations
>>
>>
>>
>> > Hi Everyone,
>> >  I was wondering if anyone was working on or knew of a way to optimize
>> > the
>> Zend_Date, Zend_Locale and Zend_Translate class. Currently, I've built a
>> wrapper class around Zend_Cache, that after it successfully connects to
>> Memcached, it execute the following:
>> >
>> > try {
>> >    /**
>> >     * @see Zend_Cache
>> >     */
>> >    require 'Zend/Cache.php';
>> >
>> >    $this->_cache = Zend_Cache::factory('Core', $config['type'],
>> > $frontend,
>> $backend);
>> >    if ($frontend['caching'] && $this->_cache->save('test', 'test')) {
>> >        /**
>> >         * @see Zend_Db_Table_Abstract
>> >         */
>> >        require 'Zend/Db/Table/Abstract.php';
>> >        Zend_Db_Table_Abstract::setDefaultMetadataCache($this->_cache);
>> >
>> >        /**
>> >         * @see Zend_Locale
>> >         */
>> >        require 'Zend/Locale.php';
>> >        Zend_Locale::setCache($this->_cache);
>> >
>> >        /**
>> >         * @see Zend_Translate
>> >         */
>> >        require 'Zend/Translate.php';
>> >        Zend_Translate::setCache($this->_cache);
>> >
>> >        /**
>> >         * @see Zend_Date
>> >         */
>> >        require 'Zend/Date.php';
>> >        Zend_Date::setOptions(array('cache'=>$this->_cache));
>> >    }
>> >    else {
>> >        $this->_cache = null;
>> >    }
>> > }
>> > catch (Exception $e) {
>> > }
>> >
>> > I've been benchmarking and optimizing my app using xdebug and
>> > kcachegrind
>> and this series of require commands (to utilize APC opcode caching and
>> avoid
>> Zend_Loader) and those caching calls consumes up a large portion of each
>> of
>> my requests. Zend_Db_Table_Abstract isn't too big of a hit considering
>> I'm
>> using Zend_Db everywhere already. Is there anyway we can, for example, on
>> the Zend_Translate_Adapter_Gettext class, to use the gettext() extension
>> if
>> it's built-in? I'm suspecting most of the performance hit is from loading
>> the Locale data from the XML files. Memcached will take over after the
>> initial request, but there are a lot of require_once calls occurring
>> throughout these classes.
>> >
>> > I'm curious what others are up to and have seen.
>> >
>> > Thanks.
>> >
>> > -Justin
>> >
>>
>>

[Vincent de Lau]

Hi all,

Today I was working on setting my default locale etcetera. I'm developing a website in Dutch and want to use Zend_Locale & co. to format my dates and numbers.

In the bootstrap I tried setting some default, but Zend_Locale::setDefault() doesn't do what I was expecting. It sets a fallback for when auto discovery fails, but will not prevent Zend_Locale from trying to auto discover.

Then I looked if I could do Zend_Date::setOptions(array('locale' => $locale)) like you can do with Zend_Locale_Format::setOptions(array('locale' => $locale)). That isn't supported either.

On IRC, Ben Scholzen directed me to a patch he posted on JIRA [1]. Thomas has made a comment that this should be fixed, by somehow reusing setDefault.

I would suggest to refactor this to be able to disable auto discovery or to choose a preferred way to perform auto discovery.

I also think that a close look should be taken at some of the other component to better streamline default Zend_Locale usage.

Kind regards,

Vincent de Lau
 vincent@...


[1] http://framework.zend.com/issues/browse/ZF-2876

[thomasW]

Vincent,

first the issue is a new feature and not a bug.
second, what you propose is a BC break which will not be implemented within
minor releases. There is a restrictive no-no for BC breaks.
third, there is already a solution for the ones which want to have such a
feature and can live with patching the framework, living with BC breakes.

Please read the issues and their comments.
As mentioned this new feature will be implemented but not until 1.6 as all
other components using Zend_Locale have to support this.

So please be patient for the official release or use the available
inofficial patches.

Greetings
Thomas Weidner, I18N Team Leader, Zend Framework
http://www.thomasweidner.com

----- Original Message -----
From: "Vincent de Lau" <vincent@...>
To: <fw-i18n@...>
Sent: Wednesday, June 18, 2008 3:19 PM
Subject: [fw-i18n] Problems with defaults and options in Zend_Locale,
Zend_Date & co.


Hi all,

Today I was working on setting my default locale etcetera. I'm developing a
website in Dutch and want to use Zend_Locale & co. to format my dates and
numbers.

In the bootstrap I tried setting some default, but Zend_Locale::setDefault()
doesn't do what I was expecting. It sets a fallback for when auto discovery
fails, but will not prevent Zend_Locale from trying to auto discover.

Then I looked if I could do Zend_Date::setOptions(array('locale' =>
$locale)) like you can do with Zend_Locale_Format::setOptions(array('locale'
=> $locale)). That isn't supported either.

On IRC, Ben Scholzen directed me to a patch he posted on JIRA [1]. Thomas
has made a comment that this should be fixed, by somehow reusing setDefault.

I would suggest to refactor this to be able to disable auto discovery or to
choose a preferred way to perform auto discovery.

I also think that a close look should be taken at some of the other
component to better streamline default Zend_Locale usage.

Kind regards,

Vincent de Lau
 vincent@...


[1] http://framework.zend.com/issues/browse/ZF-2876