Creating a PEAR mirror

[Tim Jackson-2]

Hi

At work I and some colleagues use PEAR quite heavily including doing quite
a lot of fresh installs. Although we have a caching proxy locally to try
to avoid unnecessary downloads, I think we still end up fetching quite a
lot from the master server. To reduce our demands on the PEAR master
server I was wondering about creating a mirror of some sort. Some questions:

a) Who do we need to contact to get an rsync or similar account?

b) How big (in file size) is the complete mirror?

c) Would a private mirror be OK, or only one that was publically accessible?

d) Regardless of the answer to (c), is there any demand for a second
public European mirror (there is already Christian's in Germany)? If so,
is there any estimate of what that would involve in terms of resources
(bandwidth)?

Thanks

Tim

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Christian Weiske]

Hello Tim,


> At work I and some colleagues use PEAR quite heavily including doing
> quite a lot of fresh installs. Although we have a caching proxy
> locally to try to avoid unnecessary downloads, I think we still end
> up fetching quite a lot from the master server. To reduce our demands
> on the PEAR master server I was wondering about creating a mirror of
> some sort. Some questions:
>
> a) Who do we need to contact to get an rsync or similar account?
We are in the process of setting up an official mirror guide
http://pear.php.net/bugs/bug.php?id=13886
but have not come very far yet.

> b) How big (in file size) is the complete mirror?
1.8 GiB

> c) Would a private mirror be OK, or only one that was publically
> accessible?
Since we don't have a procedure to handle public mirrors, private ones
are even more away :/

> d) Regardless of the answer to (c), is there any demand for a second
> public European mirror (there is already Christian's in Germany)? If
> so, is there any estimate of what that would involve in terms of
> resources (bandwidth)?
The mirrors (mine at least) are rarely used. So currently, bandwidth
requirements are pretty low, except the bandwidth required for syncing.
Having mirrors is good anyway, so I would not say no.

However, first we have to solve some problems including how to make
sure mirrors are white ones, e.g.
http://www.heise-online.co.uk/news/Linux-package-management-systems-not-completely-secure--/111103


--
Regards/Mit freundlichen Grüßen
Christian Weiske

-= Geeking around in the name of science since 1982 =-

[Helgi Þormar Þorbjörnsson-2]

On Mon, Jul 21, 2008 at 9:17 AM, Christian Weiske <cweiske@...> wrote:

>> d) Regardless of the answer to (c), is there any demand for a second
>> public European mirror (there is already Christian's in Germany)? If
>> so, is there any estimate of what that would involve in terms of
>> resources (bandwidth)?
> The mirrors (mine at least) are rarely used. So currently, bandwidth
> requirements are pretty low, except the bandwidth required for syncing.
> Having mirrors is good anyway, so I would not say no.

That's mostly because we don't include a Geo detection - people have
to explicitly say, "I want to use the DE mirror"
I've been pondering if we could make say "a fastest mirror" solution -
not sure how yum does it - official mirror list file and does some geo
magic, btw it´s a addon to yum - was thinking about something similar
for pear, opt in solution.

That should make our mirrors more widely used tho I can´t say we´re
being hammered enough on the actual pear server to think of it as we
need more resources, I see mirrors atm as something to make the
response time for end users.

> However, first we have to solve some problems including how to make
> sure mirrors are white ones, e.g.
> http://www.heise-online.co.uk/news/Linux-package-management-systems-not-completely-secure--/111103

yeah security was one of the reasons why we haven't been spreading the
mirror idea.
thanks for the link, I'll have a peak at it.

- Helgi

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Tim Jackson-2]

Christian Weiske wrote:

>> c) Would a private mirror be OK, or only one that was publically
>> accessible?
> Since we don't have a procedure to handle public mirrors, private ones
> are even more away :/

It seems that most of the problems (policy, security, auto-discovery etc.)
are unique to public mirrors though? An rsync source is all that's
required for a private mirror.

> However, first we have to solve some problems including how to make
> sure mirrors are white ones, e.g.
> http://www.heise-online.co.uk/news/Linux-package-management-systems-not-completely-secure--/111103

Again, this only applies to public mirrors really. There are other
problems implicit in that for PEAR too, like signing of packages (not
routinely done), signing of metadata (I'm not sure this is even possible
in PEAR) etc.

Tim


--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Christian Weiske]

Hello Tim,


> > Since we don't have a procedure to handle public mirrors, private ones
> > are even more away :/
>
> It seems that most of the problems (policy, security, auto-discovery etc.)
> are unique to public mirrors though? An rsync source is all that's
> required for a private mirror.
You are right with that statement.
I just say that private mirrors are below public mirrors on our TODO.


--
Regards/Mit freundlichen Grüßen
Christian Weiske

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Igor Feghali-4]

> Again, this only applies to public mirrors really. There are other problems
> implicit in that for PEAR too, like signing of packages (not routinely
> done), signing of metadata (I'm not sure this is even possible in PEAR) etc.

how about download statistics ? do we already have any mechanism to
merge together the statistics from all mirrors ?

regards,
iGor.

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Christian Weiske]

Hello Igor,


> how about download statistics ? do we already have any
> mechanism  to
> merge together the statistics from all mirrors ?
Not yet, no.

--
Regards/Mit freundlichen Grüßen
Christian Weiske

-= Geeking around in the name of science since 1982 =-

[Igor Feghali-4]

Hello Christian,

On Wed, Jul 23, 2008 at 12:46 PM, Christian Weiske <cweiske@...> wrote:
> Not yet, no.

so IMHO that should be the 1st concern before spreading public
mirrors... otherwise our packages will start a big drop in the
download curve, which makes me worried. or isnt that as bad as i think
it is ?

regards,
iGor.

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Christian Weiske]

Hello Igor,


>>> how about download statistics ? do we already have any
>>> mechanism  to
>>> merge together the statistics from all mirrors ?  

> > Not yet, no.

> so IMHO that should be the 1st concern before spreading public
> mirrors... otherwise our packages will start a big drop in the
> download curve, which makes me worried. or isnt that as bad as i think
> it is ?

At least my mirror is rarely used. This month, I had 3 downloads of
PEAR 1.7.2, 3 Net_Socket ones and two Net_SMTP, HTTP_Request and
XML_Parser. All other ones have been downloaded once only.

Mirror usage goes up as soon as the main server is not reachable, which
happened two or three times this year - and which was the main reason
to set them up.

I can't speak for Josh, though.

--
Regards/Mit freundlichen Grüßen
Christian Weiske

-= Geeking around in the name of science since 1982 =-

[Kevin van Zonneveld-2]

On Thu, Jul 24, 2008 at 8:43 AM, Christian Weiske <cweiske@...> wrote:
> Mirror usage goes up as soon as the main server is not reachable, which
> happened two or three times this year - and which was the main reason
> to set them up.

How is the failovermechanism implemented?
Do you just change (low TTL) DNS records or is there a more elegant
technique in place?

I'm asking because maybe my company could also help out.

--
Met vriendelijke groet / Kind regards,

Kevin van Zonneveld
http://kevin.vanzonneveld.net

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Christian Weiske]

Hello Kevin,


> > Mirror usage goes up as soon as the main server is not reachable, which
> > happened two or three times this year - and which was the main reason
> > to set them up.
>
> How is the failovermechanism implemented?
> Do you just change (low TTL) DNS records or is there a more elegant
> technique in place?

No, it's far away from elegant:
If you see that pear.php.net times out, you can manually switch the mirror to (de|us).pear.php.net via "pear config-set preferred_mirror ..". And the IRC channel gets its topic changed :)

--
Regards/Mit freundlichen Grüßen
Christian Weiske

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Helgi Þormar Þorbjörnsson-2]

On Thu, Jul 24, 2008 at 6:43 AM, Christian Weiske <cweiske@...> wrote:
> Hello Igor,
>
>
>>>> how about download statistics ? do we already have any
>>>> mechanism  to
>>>> merge together the statistics from all mirrors ?
>
>> > Not yet, no.

Hmm I thought Greg implemented a stats merging script and all that,
maybe it was only for Josh if you don't have it ?
Hrmm Josh do you know the status on that ?

Regards
Helgi

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Igor Feghali-4]

Hello Christian,

On Thu, Jul 24, 2008 at 3:43 AM, Christian Weiske <cweiske@...> wrote:
> Mirror usage goes up as soon as the main server is not reachable, which
> happened two or three times this year - and which was the main reason
> to set them up.

yeh, but as far as I could understand we are now talking about balance
of load between main server and mirrors, aren't we ?
That is a new scenario where statistics is a big concern IMHO.

best regards,
iGor.

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[JoshuaEichorn]

Christian Weiske wrote:

> Hello Igor,
>
>
>  
>>>> how about download statistics ? do we already have any
>>>> mechanism  to
>>>> merge together the statistics from all mirrors ?  
>>>>        
>
>  
>>> Not yet, no.
>>>      
>
>  
>> so IMHO that should be the 1st concern before spreading public
>> mirrors... otherwise our packages will start a big drop in the
>> download curve, which makes me worried. or isnt that as bad as i think
>> it is ?
>>    
>
> At least my mirror is rarely used. This month, I had 3 downloads of
> PEAR 1.7.2, 3 Net_Socket ones and two Net_SMTP, HTTP_Request and
> XML_Parser. All other ones have been downloaded once only.
>
> Mirror usage goes up as soon as the main server is not reachable, which
> happened two or three times this year - and which was the main reason
> to set them up.
>
> I can't speak for Josh, though.
>
>  

My mirror usage is also low, it was down for 24 hours because of an ip
change and 1 person noticed

-josh

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[JoshuaEichorn]

Helgi Þormar Þorbjörnsson wrote:

> On Thu, Jul 24, 2008 at 6:43 AM, Christian Weiske <cweiske@...> wrote:
>  
>> Hello Igor,
>>
>>
>>    
>>>>> how about download statistics ? do we already have any
>>>>> mechanism  to
>>>>> merge together the statistics from all mirrors ?
>>>>>          
>>>> Not yet, no.
>>>>        
>
> Hmm I thought Greg implemented a stats merging script and all that,
> maybe it was only for Josh if you don't have it ?
> Hrmm Josh do you know the status on that ?
>
> Regards
> Helgi
>
>  

My mirror has a download_stats.php script that is hit by the main server
on a regular basis (looks to be weekly) to update the stats.  Its been
broken for the last month but should be working fine again.

-josh



--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Helgi Þormar Þorbjörnsson-2]

On Thu, Jul 24, 2008 at 8:27 PM, Joshua Eichorn <josh@...> wrote:

> My mirror has a download_stats.php script that is hit by the main server on
> a regular basis (looks to be weekly) to update the stats.  Its been broken
> for the last month but should be working fine again.

Christian are you sure your server doesn't have it ?

- Helgi

--
PEAR Development Mailing List (http://pear.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[Christian Weiske]

Hello Helgi,


> > My mirror has a download_stats.php script that is hit by the main
> > server on a regular basis (looks to be weekly) to update the
> > stats.  Its been broken for the last month but should be working
> > fine again.
>
> Christian are you sure your server doesn't have it ?

I have it, too :)

--
Regards/Mit freundlichen Grüßen
Christian Weiske

-= Geeking around in the name of science since 1982 =-