Making kio_http more accessible to RESTful clients

[Josef Spillner]

Hello,

to make the creation of clients to RESTful services possible, applications
need access to HTTP response codes.
Since I'm not too familiar with the KIO code, I'm asking here first before
putting in my patches from playground/libs/webtech.

- slavebase: Why is metadata discarded in the case of error()? It seems more
useful to me to keep it.
- deletejob: It surely is an error that it creates a subjob and then dismisses
the subjob's metadata?
- kio_http: Its desktop file only indicates reading capabilities, whereas
writing and deleting is also supported. What's the reason behind this
restriction?
- kio: What is the rationale for having specialised methods like
KIO::http_post when all the functionality they include could be expressed by
the existing methods? Such methods should be removed or put into convenience
classes.

For KDE 5, my wishlist includes splitting the WebDAV code from the HTTP code,
to make the class more maintainable and more useful for other HTTP-based
protocols. Also, the API naming convention regarding metadata is very lousy,
it is hard to differentiate between outgoing and incoming data, especially in
KIO::Job::{setM,m}etaData().

Josef

[Thiago Macieira]

Josef Spillner wrote:
>- kio: What is the rationale for having specialised methods like
>KIO::http_post when all the functionality they include could be
> expressed by the existing methods? Such methods should be removed or
> put into convenience classes.

POST is not supported by any other method. It's an HTTP-specific method,
handled by the "special" call.

--
  Thiago Macieira  -  thiago (AT) macieira.info - thiago (AT) kde.org
    PGP/GPG: 0x6EF45358; fingerprint:
    E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358

[Josef Spillner]

Am Sonntag 05 Oktober 2008 21:16:20 schrieb Thiago Macieira:
> POST is not supported by any other method. It's an HTTP-specific method,
> handled by the "special" call.

We either want to offer an abstraction layer with KIO, or we don't.
From what I can see data can either be put somewhere, or it can be given to a
service to do something with it. In this regard, a HTTP POST is highly
similar to sending mail by SMTP or calling a SOAP operation. They all do
something with data without any guarantee to give it back to you.

Josef

[Thiago Macieira]

Josef Spillner wrote:

>Am Sonntag 05 Oktober 2008 21:16:20 schrieb Thiago Macieira:
>> POST is not supported by any other method. It's an HTTP-specific
>> method, handled by the "special" call.
>
>We either want to offer an abstraction layer with KIO, or we don't.
>From what I can see data can either be put somewhere, or it can be given
> to a service to do something with it. In this regard, a HTTP POST is
> highly similar to sending mail by SMTP or calling a SOAP operation.
> They all do something with data without any guarantee to give it back
> to you.

SOAP's HTTP transport is always POST. But that's just the transport: even
for obtaining data it uses POST.

I would argue that SMTP sending (even according to your description) is
"put", not "post". It sends something and maybe it'll give you a
confirmation and an identification. That's all.

HTTP's POST is a special kind of operation that both uploads data and then
downloads it. It's not something you'll easily find in other protocols.
That's why it's "http_post" in KIO.

--
  Thiago Macieira  -  thiago (AT) macieira.info - thiago (AT) kde.org
    PGP/GPG: 0x6EF45358; fingerprint:
    E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358

[Josef Spillner]

Am Montag, 6. Oktober 2008 08:12:35 schrieb Thiago Macieira:
> SOAP's HTTP transport is always POST. But that's just the transport: even
> for obtaining data it uses POST.

Wrong. Please consult part 2 of the specification where it clearly mandates
the implementation of both POST and GET. But in the context of this
discussion, it doesn't matter. What matters is that SOAP operations process
data instead of working on locatable resources.

> I would argue that SMTP sending (even according to your description) is
> "put", not "post". It sends something and maybe it'll give you a
> confirmation and an identification. That's all.

The semantics of PUT is that a resource is stored and can be retrieved from
the same location (barring a 301-PUT where it can be retrieved from another
location).

The semantics of POST is that some processing is done with the data, not
necessarily that it is stored. A 204-POST processes without giving feedback
other than a confirmation. Exactly what you described above, hence SMTP
follows POST semantics and not PUT semantics (especially since essentially all
MTAs modify messages by design, adding Received headers and whatnot).

> HTTP's POST is a special kind of operation that both uploads data and then
> downloads it. It's not something you'll easily find in other protocols.
> That's why it's "http_post" in KIO.

POST semantics doesn't imply "downloading data". The result might refer to a
newly created resource, but it doesn't have to. Please read section 9.5 of the
relevant specification.

At an abstract level, to avoid confusion, my suggestion is to use a naming
which reflects the difference of storing vs. processing. That is all.

Josef

[Thiago Macieira]

On Monday 06 October 2008 09:02:19 Josef Spillner wrote:
> At an abstract level, to avoid confusion, my suggestion is to use a naming
> which reflects the difference of storing vs. processing. That is all.

When KIO was designed, back in KDE 2 days, the semantic was that which I
described. There was no REST those days.

And SMTP isn't really a filesystem, so someone chose to implement the sending
of email using the standard "put" operation. You may notice that there's no
"get" for SMTP.

In any case, libkio went mostly untouched for the KDE 4 porting effort. Aside
from the porting to Qt 4 and KDE 4 lower libraries, plus a cleaning up that I
did, nothing happened.

Given that track record, I expect KDE 5's KIO to be exactly like that again.
Mostly because it just works and it isn't "sexy" to touch the I/O internals.

--
  Thiago Macieira  -  thiago (AT) macieira.info - thiago (AT) kde.org
    PGP/GPG: 0x6EF45358; fingerprint:
    E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358

On Sunday 05 October 2008, Josef Spillner wrote:
> Hello,
>
> to make the creation of clients to RESTful services possible, applications
> need access to HTTP response codes.
> Since I'm not too familiar with the KIO code, I'm asking here first before
> putting in my patches from playground/libs/webtech.
>
> - slavebase: Why is metadata discarded in the case of error()? It seems more
> useful to me to keep it.

The point of metadata in slaves is to use it, or to send it back to the application;
at the time of error(), there is no "use it" to be done anymore (the current operation is over),
and there hasn't been a need to sent it to the app until now, since the job is over,
but I guess this could be changed.

> - deletejob: It surely is an error that it creates a subjob and then dismisses
> the subjob's metadata?

Arguably, yes. In copyjob we do   m_incomingMetaData += kiojob->metaData();
when kiojob emits result, I wouldn't be opposed to deletejob doing the same.

> - kio_http: Its desktop file only indicates reading capabilities, whereas
> writing and deleting is also supported. What's the reason behind this
> restriction?

You mean .protocol file? There's a webdav.protocol which defines this.
http.protocol doesn't since you can't write or delete over a http url afaik.

> - kio: What is the rationale for having specialised methods like
> KIO::http_post when all the functionality they include could be expressed by
> the existing methods? Such methods should be removed or put into convenience
> classes.

Well it's a convenience method, how is that different from a convenience class?
"Expressed by existing methods" -- well, not really, if you take into account
the big security check in that method (precheckHttpPost).

> For KDE 5, my wishlist includes splitting the WebDAV code from the HTTP code,
> to make the class more maintainable and more useful for other HTTP-based
> protocols. Also, the API naming convention regarding metadata is very lousy,
> it is hard to differentiate between outgoing and incoming data, especially in
> KIO::Job::{setM,m}etaData().

Yep, feel free to clarify the docs. The implementation is rather clear at that place,
e.g. KIO::Job::setMetaData is outgoing, it's metadata that will be sent to the slave,
while KIO::Job::metaData/queryMetaData is the incoming metadata from job.
A bit unusual that the getter and setter are not symmetric indeed, although it
matches the way metadata is used in 99% of the cases  (there's "outgoingMetaData"
for the case of querying the metadata that we have set previously, but that's rare...).
I'm not sure that setOutgoingMetaData() would be considered clearer than setMetaData()
in the common use case...

--
David Faure, faure@..., sponsored by Trolltech to work on KDE,
Konqueror (http://www.konqueror.org), and KOffice (http://www.koffice.org).

[Josef Spillner]

Am Montag, 6. Oktober 2008 23:46:37 schrieb David Faure:
> The point of metadata in slaves is to use it, or to send it back to the
> application; at the time of error(), there is no "use it" to be done
> anymore (the current operation is over), and there hasn't been a need to
> sent it to the app until now, since the job is over, but I guess this could
> be changed.

I think most of the metadata can indeed be discarded, but what about e.g. the
error codes of the underlying protocol which can give further explanation for
the reason of a job termination? This could be useful information to
applications which otherwise would have to use QHttp or their own handlers
just for this missing piece in the puzzle.

> > - deletejob: It surely is an error that it creates a subjob and then
> > dismisses the subjob's metadata?
>
> Arguably, yes. In copyjob we do   m_incomingMetaData += kiojob->metaData();
> when kiojob emits result, I wouldn't be opposed to deletejob doing the
> same.

Ok.

> > - kio_http: Its desktop file only indicates reading capabilities, whereas
> > writing and deleting is also supported. What's the reason behind this
> > restriction?
>
> You mean .protocol file? There's a webdav.protocol which defines this.
> http.protocol doesn't since you can't write or delete over a http url
> afaik.

WebDAV is an extension to HTTP, but HTTP already allows deletion on its own.
From what I know the first web browsers even had a GUI for that. Right-click
on image and delete, scary thought :-)
From what I can see there should be no side effects when adding
writing/deletion to the .protocol file, but then again I don't know much about
the implications and areas of use of this information.

> > - kio: What is the rationale for having specialised methods like
> > KIO::http_post when all the functionality they include could be expressed
> > by the existing methods? Such methods should be removed or put into
> > convenience classes.
>
> Well it's a convenience method, how is that different from a convenience
> class? "Expressed by existing methods" -- well, not really, if you take
> into account the big security check in that method (precheckHttpPost).

If it's such an important security check, a bit of text for the API docs
wouldn't be bad :)
From what I can see the check includes things which should be handled at a
different level, as they're not intrinsic to the protocol (or any protocol),
e.g. the KAuthorized check. Something like setPreprocessingHandler() comes to
my mind. Note that my intention is not to rewrite KIO by myself, it is to
gather requirements and design decisions which could be improved at some point
when the information about what actually needs to be done is available
somewhere. Kind of like a high-level TODO file.

Josef

[Andreas Hartmetz-2]

Am Sonntag 05 Oktober 2008 18:49:52 schrieb Josef Spillner:

> For KDE 5, my wishlist includes splitting the WebDAV code from the HTTP
> code, to make the class more maintainable and more useful for other
> HTTP-based protocols. [...]
>
There is a lot of similarity, and honestly, the internal complexity of the
HTTP protocol implementation alone much worse than the added complexity of
WebDAV which actually *is* an extension of HTTP. I don't see much benefit in
splitting them now that the code is written.

If there are any volunteers - factoring out the disk caching code and the
authentication code each into a class of their own should help. After that
the rest should be easier as well. Smaller, more specialized data structures
would help too.