Architecture the new DataSystem!

[Jaroslav Tulach]

Hi.
During my performance work I found out that major problem when expanding
folders and files is the amount of work done by DataSystems. Especially the
style of DataLoader registration is heavily inefficient. We need scalability!
We need a change! We need it for 6.5! Please help me architecture the change:

http://wiki.netbeans.org/FitnessAgainstForgetfulness

Add your comments into the wiki, send them here, add them into the issue.
Thanks and please send them by next week. I want to integrate by end of May.

-jst

[Emilian Bold]

Data Loaders are the dark swamps of NB Platform I never go to...

Now, regarding this change, you seem to already have a patch ready so
you don't need help with the architecture of the new Data System,
you've pretty much decided. You also want this in one week with an
integration target of end of May so again, something really quick
which could bring little input.

I'm ranting here as the title of the email and tone made me think:
-"Gee, we have the opportunity to engineer something new here. Who
knows what great ideas will show up. Especially regarding Data System,
the dreaded place".

But then I saw the 1 week deadline and the diff on the wiki...

Emilian

On Wed, May 14, 2008 at 1:07 AM, Jaroslav Tulach
<Jaroslav.Tulach@...> wrote:

> Hi.
> During my performance work I found out that major problem when expanding
> folders and files is the amount of work done by DataSystems. Especially the
> style of DataLoader registration is heavily inefficient. We need scalability!
> We need a change! We need it for 6.5! Please help me architecture the change:
>
> http://wiki.netbeans.org/FitnessAgainstForgetfulness
>
> Add your comments into the wiki, send them here, add them into the issue.
> Thanks and please send them by next week. I want to integrate by end of May.
>
> -jst
>

--
Emilian Bold
[phone]
http://www.emilianbold.ro

Java and NetBeans Platform-loving consulting services from Timisoara, Romania.

[Marian Petras]

Jarda,

first, I agree with Emilian that this is not about a new architecture -
it is about one small change having big impact on performance.

I like the concept of restriction of number of DataLoaders used when a
DataObject is being created for a given file and the proposal is just
logical. But when looking at the diff, the structure of the XML layer
and naming of the XML elements does not seem logical to me:

     "Loaders"
         <MIME-type>
             "Factories"
                 <list of loaders to try>
             "Actions"
                 <list of actions>

What does the list of actions represent? I think this is 100% functional
in case of 1:1 mapping between MIME-type and DataObject types. How is it
exactly used in other cases?

The following would seem more logical to me:

     "MIME-types"
         <MIME-type>
             "Loaders"
                 <list of loaders to try>

     "DataObject-types"
         <DataObject type id>
             "Actions"
                 <list of actions>

I know that your proposed change does not make this any worse. I just
wanted to ask about it now that we are discussing related changes.

Marián


Jaroslav Tulach wrote:

> Hi.
> During my performance work I found out that major problem when expanding
> folders and files is the amount of work done by DataSystems. Especially the
> style of DataLoader registration is heavily inefficient. We need scalability!
> We need a change! We need it for 6.5! Please help me architecture the change:
>
> http://wiki.netbeans.org/FitnessAgainstForgetfulness
>
> Add your comments into the wiki, send them here, add them into the issue.
> Thanks and please send them by next week. I want to integrate by end of May.
>
> -jst

[Jaroslav Tulach]

Dne Thursday 15 May 2008 16:15:10 Emilian Bold napsal(a):

> Data Loaders are the dark swamps of NB Platform I never go to...
>
> Now, regarding this change, you seem to already have a patch ready so
> you don't need help with the architecture of the new Data System,
> you've pretty much decided. You also want this in one week with an
> integration target of end of May so again, something really quick
> which could bring little input.
>
> I'm ranting here as the title of the email and tone made me think:
> -"Gee, we have the opportunity to engineer something new here. Who
> knows what great ideas will show up. Especially regarding Data System,
> the dreaded place".
>
> But then I saw the 1 week deadline and the diff on the wiki...

Thanks for rant and here is one back:

Should you have read the diff, you would find out that it is not diff of the
implementation but the way people will use the new API. Next time spend more
time reading than complaining. It will be more useful.

Sorry for the 1 week deadline. I know common is to have 2 weeks. However if we
want this to get to 6.5 things need to happen fast. The conf call will be on
Thursday, May 22, 2008 at 17.30CET, if anyone is interested, let me know.

-jst

> On Wed, May 14, 2008 at 1:07 AM, Jaroslav Tulach
>
> <Jaroslav.Tulach@...> wrote:
> > Hi.
> > During my performance work I found out that major problem when expanding
> > folders and files is the amount of work done by DataSystems. Especially
> > the style of DataLoader registration is heavily inefficient. We need
> > scalability! We need a change! We need it for 6.5! Please help me
> > architecture the change:
> >
> > http://wiki.netbeans.org/FitnessAgainstForgetfulness
> >
> > Add your comments into the wiki, send them here, add them into the issue.
> > Thanks and please send them by next week. I want to integrate by end of
> > May.
> >
> > -jst

[Jaroslav Tulach]

Dne Friday 16 May 2008 09:48:03 Marian Petras napsal(a):

> Jarda,
>
> first, I agree with Emilian that this is not about a new architecture -
> it is about one small change having big impact on performance.
>
> I like the concept of restriction of number of DataLoaders used when a
> DataObject is being created for a given file and the proposal is just
> logical. But when looking at the diff, the structure of the XML layer
> and naming of the XML elements does not seem logical to me:
>
>      "Loaders"
>          <MIME-type>
>              "Factories"
>                  <list of loaders to try>
>              "Actions"
>                  <list of actions>
>
> What does the list of actions represent? I think this is 100% functional
> in case of 1:1 mapping between MIME-type and DataObject types. How is it
> exactly used in other cases?

I can imagine how it is done for form files. The loader is registered twice,
once in Factories/text/x-form and once in Factories/text/x-java, its actions
are however just in Actions/text/x-form. This is not really consistent, for
more complex cases, however for 90% of the simple ones it is quite OK.

> The following would seem more logical to me:
>
>      "MIME-types"
>          <MIME-type>
>              "Loaders"
>                  <list of loaders to try>
>
>      "DataObject-types"
>          <DataObject type id>
>              "Actions"
>                  <list of actions>
>
> I know that your proposed change does not make this any worse. I just
> wanted to ask about it now that we are discussing related changes.

Yes, this is more logical, however it is a bit harder to find. As the actions
and loader registrations are really not next to each other. Indeed we cannot
change existing registrations. However if this solution shall be preferred,
then I would probably suggest to use /LoaderFactories/mime/type/x.instance

Anyway I do not know what is better. Let's discuss this on the meeting on
Thursday.
-jst

PS: Whoever wants to join, send an email to me. I'll provide conf call info.

[Jaroslav Tulach]

Thanks for your review. I have updated
http://wiki.netbeans.org/FitnessAgainstForgetfulness
I'll fix all TCRs and next week, I'll integrate the change, together with
change in registration of all the loaders.

-jst

[Petr Nejedly]

Jaroslav Tulach napsal(a):
> Thanks for your review. I have updated
> http://wiki.netbeans.org/FitnessAgainstForgetfulness
> I'll fix all TCRs and next week, I'll integrate the change, together with
> change in registration of all the loaders.

Please notice the broken javadoc of DataObject.Factory.findDataObject
in the X.diff (recognized vs. markRecognized)

--
Petr "Nenik" Nejedly, NetBeans/Sun Microsystems, http://www.netbeans.org
355/113 -- Not the famous irrational number PI, but an incredible simulation!