|
»
»
Re: GSoC ShlibMemLoad
|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 | Next > |
 |
|
|
|
Re: GSoC ShlibMemLoad
by Daniel Hans
::
Rate this Message:
2008/7/4 Andreas Kupries <andreask@...>:
> >> >> They all have only one actual function greet(). This function get pid >> >> of the process which calls it and tries to write it to stdout. >> >> libwrite_simple makes it in the easiest possible way: just uses one >> >> write sys call. >> >> libwrite_global makes it similarly, but a buffer is global now. >> >> libwrite_fun calls another function (fill_str to fill buffer with data) >> >> These three libraries seems to be working fine. >> >> The fourth library libprintf tries to make the same thing easier and >> >> use printf instead of write. Anyway, it does not work. I get a >> >> segmentation fault (the same happens when I substitute printf for any >> >> other functions from stdio.h like puts). >> >> >> I do not know what may be wrong but I will try to work on it. Maybe >> >> the problem is in function load_needed_objects(Obj_Entry *) which is >> >> called by elf_dlopen, but I still do not know why it works for the >> >> first three libraries. They use functions from unistd.h. >> >> >I can't say at the moment. It seems that you haven't committed all the >> >changes yet, so I have can't see what was done. >> >> >For example, it is not clear if you are using the OS dlsym() already, or >> >not. >> >> >Best guess I have right now, is that it might be that your non-working >> >library depends on some other library FOO instead of just libc. And while >> >libc symbols are present the FOO is not loaded (yet) and thus its symbols >> >are not present either? >> >> As far as I know printf function is in libc as well. I also tried to >> compile libprintf.c to a program and run it with strace. The only >> library which was loaded was libc. Function load_needed_objects said >> it needed only libc... >> When I look at the output of elf_dlopen it seems that printf >> symbol is founded. >> I also wrote another simple library: libcos.c. It calls cos function >> defined in math.h. This library has to be linked with libm (-lm). And >> rtld copes with this library... >> Another simple library libstrcpy whch uses strcpy also works. >> So I still have no idea what is the difference between libprintf and >> those other libraries which works. > > While walking to the office I managed to come up with a number of additional > ideas ... > > printf is different from the other functions in two respects. > > It takes a variable number of arguments, > and it operates at a higher IO level than write and consorts. > > Both might be a possible source of the problem I guess. > Although I have no idea of how that could be. > > Things to consider: > > - Does using 'sprintf()' (+write) work ? > sprintf is also varargs, but doesn't do any I/O > > - Does 'fprintf()' work ? > It is like printf, i.e. varargs and highlevel I/O > However it explicitly takes an output argument. > printf (...) is like fprintf (stdout, ...). fprintf does not work > - Does fflush() work? > Highlevel I/O, but no varargs. It is also outside > of the printf family of functions in general. fflush surprisingly works... Anyway, the problem is not in varargs, because I also tried with functions like putchar, puts, etc and they all does not work. > > To anybody, are there varargs functions in libc outside of the printf family > which could be used to probe the problem further? > > > General background for the Tcl-core guys just coming in on the thread ... > > This is Google Summer of Code Project 'Loading a shared library from > memory'. > The svn repository for all relevant sources is at > http://svn.assembla.com/svn/gsoc2008-tcl_dynamic_libraries/ > > Daniel is currently experimenting with FreeBSD's (*) runtime ld (aka > "rtld") > This is in essence the "ld.so" used to load regular libraries from disk, > and > > Daniel's basic idea is that we should be able to modify this in only a few > places to get it to process a memory block holding a library, and hook to > other > parts of Tcl and the OS. > > He is currently testing the basic setup, getting to know the sources and > its > structure, and getting it to work for regular loading. > > From the above, most of the test libraries can be loaded fine, some of them > crash however and he is working on finding out why. > > > (*) While glibc has the same functionality and sources available it is under > GPL, which does not mesh nicely with our BSD world. FreeBSD's code and > license does. > > -- > Andreas Kupries <andreask@...> > Developer @ http://www.ActiveState.com > Tel: +1 778-786-1122 > > ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
|
|
|
Re: GSoC ShlibMemLoad
by Daniel Hans
::
Rate this Message:
2008/7/5 Andreas Kupries <akupries@...>:
> >> > While walking to the office I managed to come up with a number of additional >> > ideas ... >> > >> > printf is different from the other functions in two respects. >> > >> > It takes a variable number of arguments, >> > and it operates at a higher IO level than write and consorts. >> > >> > Both might be a possible source of the problem I guess. >> > Although I have no idea of how that could be. >> > >> > Things to consider: >> > >> > - Does using 'sprintf()' (+write) work ? >> > sprintf is also varargs, but doesn't do any I/O >> sprintf does not work >> > >> > - Does 'fprintf()' work ? >> > It is like printf, i.e. varargs and highlevel I/O >> > However it explicitly takes an output argument. >> > printf (...) is like fprintf (stdout, ...). >> fprintf does not work >> > - Does fflush() work? >> > Highlevel I/O, but no varargs. It is also outside >> > of the printf family of functions in general. >> fflush surprisingly works... >> Anyway, the problem is not in varargs, because I also tried with >> functions like putchar, puts, etc and they all does not work. > > Hm ... All these functions have lots of state ... > > > Some leading questions ... > > You are using 'printf' in rtld itself to print log/debug messages, and > it works there. What is the address of this 'printf' function ? > > Is the address of the 'printf' function seen by the loaded > test-library the same ? (**) > > If not, why? log/debug messages uses printf from libc.so which was loaded at the beginning of rtld execution by the regular dynamic loader and printf in libprintf.so uses printf from libc.so loaded by load_needed_objects. > Do you remember my comments about missing symbols ? > > I.e.: The list of libraries we have in our rtld is the list of > memory-loaded libraries (ML/lib) and not of all libraries > (ALL/lib). > > And a symbol not found in ML/lib has to be looked for in > ALL/lib too ? > > What does that same distinction mean for 'load_needed_objects' ? and if our library needs another one it is loaded for the second time. Anyway, all libraries I have tested needed just libc.so. > And (especially) a library like 'libc.so' which is loaded by the OS > and thus in ALL/lib? > > > (**) In (one of) the other test libraries you are using a for-loop and > write to print the pid. If you encapsulate that into a function > FOO to generally print unsigned numbers and then put FOO into the > libprintf code you can print any number you need, like function > addresses. Inside of rtld itself it is of course possible to use > the working printf to print its address. > and have not came up with any solution I am thinking of writing an email to John Polstra who is an author of the rtld for FreeBSD. Maybe he will be able to help out. Do you think it is a good idea? Today I also wrote a simple program which generates defs.h file. Greetings, Daniel ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
|
|
|
Re: GSoC ShlibMemLoad
by Tomasz Kosiak-2
::
Rate this Message:
On Mon, Jul 7, 2008 at 5:28 AM, Andreas Kupries <akupries@...> wrote:
[...] >> Because I have already studied this whole code for quite a long time >> and have not came up with any solution I am thinking of writing an >> email to John Polstra who is an author of the rtld for >> FreeBSD. Maybe he will be able to help out. Do you think it is a >> good idea? > > Yes. Getting the help of (one of) the original authors of some code is > (in my opinion) a good idea. He should know the intricacies of the > code. As Daniel intends to contact John Polstra from FreeBSD I think that it would be easier to work with him if Daniel installs FreeBSD and will convey his experiments (and work with John) in this enviroment. I would recommend installing FreeBSD on separate machine or in some kind of virtual enviroment like VMware or VirtualBox to make simultaneous testing easy (such a FreeBSD installation could run in runlevel 3 without graphical console to limit resources requirements). >From high level perspective what we need is support in OS dynamic linker to link with some libraries loaded by our program into memory from some virtual filesystem specific to application (ex. starkit for Tcl). Such feature can benefit not only Tcl, but also others who think about single file deployments. As far as I understand MacOSX already has got this functionality and other OS-es lack it. We currently try to emulate it in Linux using code from FreeBSD. But maybe first we should try to make it work in FreeBSD. FreeBSD folks can even maybe add such capability as a system call (keep in mind that FreeBSD and MacOSX shares some source code and maybe it could be also helpful). Concerning Linux we decided to emulate loading dynlibs from memory using FreeBSD code (due to licensing issues with GPL vs BSD). But maybe we shall also contact Linux/glibc developers about including proposed functionality in OS/glibc. I don't know how we are going to handle Solaris and other platforms that Tcl is known to run fine. It is certainly beyond scope of Daniels Google Summer of Code project. But first we have to think also about Microsoft Windows where starkit/starpack deployments will benefit users the most. I have no idea how can we achieve it on Windows - I don't think we can convince/help Microsoft to add such functionality :-(. Regards, Tomasz Kosiak ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
Re: GSoC ShlibMemLoad
by Daniel Hans
::
Rate this Message:
2008/7/7 Andreas Kupries <akupries@...>:
> >> > Some leading questions ... >> > >> > You are using 'printf' in rtld itself to print log/debug messages, and >> > it works there. What is the address of this 'printf' function ? >> > >> > Is the address of the 'printf' function seen by the loaded >> > test-library the same ? (**) >> > >> > If not, why? > >> These addresses differ. I am not sure why, but I think that printf in >> log/debug messages uses printf from libc.so which was loaded at the >> beginning of rtld execution by the regular dynamic loader and printf >> in libprintf.so uses printf from libc.so loaded by >> load_needed_objects. > > Yes, they differed for me as well, and I am drawing the same conclusion. > >> > Do you remember my comments about missing symbols ? >> > >> > I.e.: The list of libraries we have in our rtld is the list of >> > memory-loaded libraries (ML/lib) and not of all libraries >> > (ALL/lib). >> > >> > And a symbol not found in ML/lib has to be looked for in >> > ALL/lib too ? >> > >> > What does that same distinction mean for 'load_needed_objects' ? > >> I think that load_needed_objects does not know anything about ALL/lib > > Yes, I agree here as well. However, I further also believe that > load_needed_objects() will have to be changed to be made aware of > ALL/lib in some kind to prevent this duplicate loading. > > I.e. let it first search for the library by itself (*), and when that > fails let it fall back to dlopen() to load the needed object from the > outside. dlopen() will then consult ALL/lib. The main point here is > that the first search should fail to locate libc.so (*), and then > gotten through dlopen(), thus ALL/lib. That way we should be able to > avoid loading it a second time. > > And the changes made to find_symdef() or other (the fallback to > dlsym()) should then take care of handling the resolution of symbols > in libc.so for us. > > (*) This implies changes to find_library(). In our use case, loading > from memory, we wish to search virtual filesystems. So we need an > API for Tcl to define search paths, which will be inside of the > starkit or starpack, and all the existing code using the LD_* > and other environment variables we should be able to discard. > >> and if our library needs another one it is loaded for the second time. >> Anyway, all libraries I have tested needed just libc.so. > > True. And most C packages for Tcl will be similar. Even so, there will > be packages which depend on more shared libraries (Example: TclDOM has > a C-based variant depending on libxml2.so). > >> > And (especially) a library like 'libc.so' which is loaded by the OS >> > and thus in ALL/lib? >> > >> > >> > (**) In (one of) the other test libraries you are using a for-loop and >> > write to print the pid. If you encapsulate that into a function >> > FOO to generally print unsigned numbers and then put FOO into the >> > libprintf code you can print any number you need, like function >> > addresses. Inside of rtld itself it is of course possible to use >> > the working printf to print its address. > >> Because I have already studied this whole code for quite a long time >> and have not came up with any solution I am thinking of writing an >> email to John Polstra who is an author of the rtld for >> FreeBSD. Maybe he will be able to help out. Do you think it is a >> good idea? > > Yes. Getting the help of (one of) the original authors of some code is > (in my opinion) a good idea. He should know the intricacies of the > code. I wrote the email today in the morning. > >> Today I also wrote a simple program which generates defs.h file. > > That note I did not understand. Can you explain a bit more? I committed make_defs.c file which automatically generates file elf-defs.h (the same as defs.h). An argument for this program should be the program itself, like ./make_defs make_defs > -- > So long, > Andreas Kupries <akupries@...> > <http://www.purl.org/NET/akupries/> > Developer @ <http://www.activestate.com/> > ------------------------------------------------------------------------------- > > > ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
Re: GSoC ShlibMemLoad
by Andreas Kupries-2
::
Rate this Message:
> >> > (**) In (one of) the other test libraries you are using a
> for-loop and > >> > write to print the pid. If you encapsulate that into a function > >> > FOO to generally print unsigned numbers and then put FOO into the > >> > libprintf code you can print any number you need, like function > >> > addresses. Inside of rtld itself it is of course possible to use > >> > the working printf to print its address. > > > >> Because I have already studied this whole code for quite a long time > >> and have not came up with any solution I am thinking of writing an > >> email to John Polstra who is an author of the rtld for > >> FreeBSD. Maybe he will be able to help out. Do you think it is a > >> good idea? > > > > Yes. Getting the help of (one of) the original authors of some code is > > (in my opinion) a good idea. He should know the intricacies of the > > code. > > I wrote the email today in the morning. > > > >> Today I also wrote a simple program which generates defs.h file. > > > > That note I did not understand. Can you explain a bit more? > > I committed make_defs.c file which automatically generates file > elf-defs.h (the same as defs.h). > An argument for this program should be the program itself, like > ./make_defs make_defs Hm. Ok, the defs.h file is the header which contains all the definitions #define Elf_Half Elf32_Half #define Elf_Addr Elf32_Addr #define Elf_Rel Elf32_Rel etc. From the contents of the generator file make_defs.c I infer that you doing this to switch the code between 32 and 64 bit ELF systems (and possibly other ). Is that right ? What are (dis)advantages of this approach compared to the alternative of using a single defs.h file and preprocessor conditionals to switch between the possibilities ? I.e have one defs.h file containing, roughly: #ifdef USE_ELF_64 /* Or whatever other name sounds more appropriate */ # define Elf_Half Elf64_Half # define Elf_Addr Elf64_Addr # define Elf_Rel Elf64_Rel [...] #else /* 32 bit / default branch */ # define Elf_Half Elf32_Half # define Elf_Addr Elf32_Addr # define Elf_Rel Elf32_Rel [...] #endif and then using it like so gcc -DUSE_ELF_64 foo.c ... // on 64bit elf systems and gcc foo.c ... // on 32bit elf systems -- Andreas Kupries <andreask@...> Developer @ http://www.ActiveState.com Tel: +1 778-786-1122 ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
Re: GSoC ShlibMemLoad
by Andreas Kupries-2
::
Rate this Message:
> On Mon, Jul 7, 2008 at 5:28 AM, Andreas Kupries <akupries@...> wrote:
> > [...] > > >> Because I have already studied this whole code for quite a long time > >> and have not came up with any solution I am thinking of writing an > >> email to John Polstra who is an author of the rtld for > >> FreeBSD. Maybe he will be able to help out. Do you think it is a > >> good idea? > > > > Yes. Getting the help of (one of) the original authors of some code is > > (in my opinion) a good idea. He should know the intricacies of the > > code. > > As Daniel intends to contact John Polstra from FreeBSD I think that it > would be easier to work with him if Daniel installs FreeBSD and will > convey his experiments (and work with John) in this enviroment. I > would recommend installing FreeBSD on separate machine or in some kind > of virtual enviroment like VMware or VirtualBox to make simultaneous > testing easy (such a FreeBSD installation could run in runlevel 3 > without graphical console to limit resources requirements). I have no big opinion here. I have never worked with FreeBSD, so I have no idea how much effort it will be to set it up. An image for some virtual machine is likely the easiest way to go for that. A quick google for 'freebsd vmware image' gave me http://people.freebsd.org/~matusita/VMware/ and http://www.thoughtpolice.co.uk/vmware/ (scroll down the second for freebsd). And there could be more. However, will Daniel's machine have the cpu-power needed to host both the freebsd image and his main system ? Also, how much time will we lose by such a switch? > >From high level perspective what we need is support in OS dynamic > linker to link with some libraries loaded by our program into memory > from some virtual filesystem specific to application (ex. starkit for > Tcl). Such feature can benefit not only Tcl, but also others who think > about single file deployments. As far as I understand MacOSX already > has got this functionality and other OS-es lack it. We currently try > to emulate it in Linux using code from FreeBSD. But maybe first we > should try to make it work in FreeBSD. FreeBSD folks can even maybe > add such capability as a system call (keep in mind that FreeBSD and > MacOSX shares some source code and maybe it could be also helpful). > Concerning Linux we decided to emulate loading dynlibs from memory > using FreeBSD code (due to licensing issues with GPL vs BSD). But > maybe we shall also contact Linux/glibc developers about including > proposed functionality in OS/glibc. In general I would welcome it if the various operating systems would support loading from memory out of the box with a (semi-)standard system call like dlopenmem() or some such. For now I would go with our work on emulating this and then use the concrete code coming out of the project to give the proposal more weight. > I don't know how we are going to handle Solaris and other platforms > that Tcl is known to run fine. It is certainly beyond scope of Daniels > Google Summer of Code project. But first we have to think also about > Microsoft Windows where starkit/starpack deployments will benefit > users the most. I have no idea how can we achieve it on Windows - I > don't think we can convince/help Microsoft to add such functionality > :-(. Solaris and other unixes, except the oldest (*), are all based on ELF. We should be able to use the project results for them as well. windows uses PE and/or COFF if I remember correctly, that would need a separate branch, yes. And I agree that MS is likely not receptive to such a proposal. -- Andreas Kupries <andreask@...> Developer @ http://www.ActiveState.com Tel: +1 778-786-1122 ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
Re: GSoC ShlibMemLoad
by Andreas Kupries-2
::
Rate this Message:
As a general note, for those interested in looking at things deeper, the project repository is available at andreask@gila:~/workbench/shmemload/gsoc2008-tcl_dynamic_libraries> svn info Path: . URL: http://svn.assembla.com/svn/gsoc2008-tcl_dynamic_libraries Repository UUID: d7ba3245-3e7b-42d0-9cd4-356c8b94b330 Revision: 6 Node Kind: directory Schedule: normal Last Changed Author: dani3l Last Changed Rev: 6 Last Changed Date: 2008-07-06 13:04:21 -0700 (Sun, 06 Jul 2008) -- Andreas Kupries <andreask@...> Developer @ http://www.ActiveState.com Tel: +1 778-786-1122 ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
Re: GSoC ShlibMemLoad
by Andreas Kupries-2
::
Rate this Message:
> >> Because I have already studied this whole code for quite a long time > >> and have not came up with any solution I am thinking of writing an > >> email to John Polstra who is an author of the rtld for > >> FreeBSD. Maybe he will be able to help out. Do you think it is a > >> good idea? > > > > Yes. Getting the help of (one of) the original authors of some code is > > (in my opinion) a good idea. He should know the intricacies of the > > code. > > I wrote the email today in the morning. Thank you. Please add us (*) to the CC: and/or To: list of your conversation with John. (*) DanielS, Tomasz, Tcl-core, and myself. -- Andreas Kupries <andreask@...> Developer @ http://www.ActiveState.com Tel: +1 778-786-1122 ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
Re: GSoC ShlibMemLoad
by Donal K. Fellows-2
::
Rate this Message:
Andreas Kupries wrote:
> Solaris and other unixes, except the oldest (*), are all based on ELF. We > should be able to use the project results for them as well. windows uses PE > and/or COFF if I remember correctly, that would need a separate branch, yes. > And I agree that MS is likely not receptive to such a proposal. To be clearer, even just doing ELF would be a good step forward as that would still tackle many platforms that are currently in use. For the others, falling back to the existing "copy out and load that" would do. (Mind you, if we could handle Windows DLLs too, that'd be great! It would cover just about everything that comes up in practice.) Donal. ------------------------------------------------------------------------- Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! Studies have shown that voting for your favorite open source project, along with a healthy diet, reduces your potential for chronic lameness and boredom. Vote Now at http://www.sourceforge.net/community/cca08 _______________________________________________ Tcl-Core mailing list Tcl-Core@... https://lists.sourceforge.net/lists/listinfo/tcl-core |
|
|
Re: GSoC ShlibMemLoad
by apw
::
Rate this Message:
|
