Max file size which can be transmitted using HttPClient

[ANEESHCH]

Hi,
I have a requirement for sending a huge stream, created by uploading a 2GB file, using fileupload to a remote file share using Http. When I'm tried with java.net.HttpURLConnection, it was failing by throwing OutOfmemory error.
1) Can I make use of HttpClient in this scenario?
2)What's the max file size which can be send using HttpClient?
3)Is it request streaming which I have to use for this purpose.

My application is running on, Websphere Application Server ND 5.1. and the current max heap size of eah JVM is 768M.


Any help is appreciated..

Thanks in advance

-A

[olegk]

On Sat, 2008-09-20 at 06:43 -0700, ANEESHCH wrote:

> Hi,
> I have a requirement for sending a huge stream, created by uploading a 2GB
> file, using fileupload to a remote file share using Http. When I'm tried
> with java.net.HttpURLConnection, it was failing by throwing OutOfmemory
> error.
> 1) Can I make use of HttpClient in this scenario?
> 2)What's the max file size which can be send using HttpClient?
> 3)Is it request streaming which I have to use for this purpose.
>
> My application is running on, Websphere Application Server ND 5.1. and the
> current max heap size of eah JVM is 768M.
>
>
> Any help is appreciated..
>
> Thanks in advance
>

HttpClient can stream out entities up to 2^63-1 (Long#MAX_VALUE) when
using content-length delimited content or unlimited when using chunk
encoding.

See the example:

http://svn.apache.org/repos/asf/httpcomponents/oac.hc3x/trunk/src/examples/UnbufferedPost.java

Hope this helps

Oleg


> -A


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[ANEESHCH]

Hi,
When I tried using UnbufferedPost, I got the following error:

[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R org.apache.commons.httpclient.ProtocolException: Unbuffered entity enclosing request can not be repeated.
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:487)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at com.ey.weblib.webdav.docstore.FileStreamer.write(FileStreamer.java:67)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at com.ey.weblib.webdav.docstore.RemoteDocFileManager.writeStream(RemoteDocFileManager.java:349)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr     R at com.ey.weblib.webdav.docstore.DocFileCoordinator.uploadAFile(DocFileCoordinator.java:394)
................................
Then tried with a custom entity class, and set the flag isRepeatable to true, I strated getting the following errror...

[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R java.io.IOException: Read error
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at java.io.FileInputStream.readBytes(Native Method)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at java.io.FileInputStream.read(FileInputStream.java:203)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at com.ey.weblib.webdav.docstore.InputStreamCustomEntity.writeRequest(InputStreamCustomEntity.java:50)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr     R  at
..................................
=========================================================

Please give me your opinion
-A

ANEESHCH wrote:

Hi,
I have a requirement for sending a huge stream, created by uploading a 2GB file, using fileupload to a remote file share using Http. When I'm tried with java.net.HttpURLConnection, it was failing by throwing OutOfmemory error.
1) Can I make use of HttpClient in this scenario?
2)What's the max file size which can be send using HttpClient?
3)Is it request streaming which I have to use for this purpose.

My application is running on, Websphere Application Server ND 5.1. and the current max heap size of eah JVM is 768M.


Any help is appreciated..

Thanks in advance

-A

[olegk]

On Sun, 2008-09-21 at 09:37 -0700, ANEESHCH wrote:
> Hi,
> When I tried using UnbufferedPost, I got the following error:
>

...

> ................................
> Then tried with a custom entity class, and set the flag isRepeatable to
> true, I strated getting the following errror...

Marking an entity as repeatable, when it is not, does not really help,
does it? Make sure the entity is capable of recreating its content
InputStream in order for it to be repeatable. See

http://hc.apache.org/httpclient-3.x/xref/org/apache/commons/httpclient/methods/FileRequestEntity.html

Oleg



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[ANEESHCH]

Hi,
I tried the following options so far.
1) Based on the unbuffered post, used
httppost.setRequestEntity(new InputStreamRequestEntity(
        new FileInputStream(file), file.length()));

It gave me the error, org.apache.commons.httpclient.ProtocolException: Unbuffered entity enclosing request can not be repeated. in the system error file.

2) Then I tried using FileRequestEntity, to make sure that content can be recreated, as well as for marking the entity as repeatable
No error was reported any where, but the file was not uploaded to the remote server.
Httpstatus code 404 was returned by the excecutemethod();

But initially, when I didn't set any value for the contenttype, it gave me an IO Exception, read error, then I set the content type as "tex/plain; UTF-8", the IO Error went away.

Please share your thoughts on this, some how I want to make this work.

<code snippet>
public void write(InputStream inStream, String url, int port, String user,
                        String password, String host) {

                PostMethod httppost = null;

                try {
                        System.out.println("Inside File Streamer: url=" + url + "; port="
                                        + port + "; user=" + user + ";password=" + password);

                        httppost = new PostMethod(url);
                        File file = new File("F:\\FileServers.txt");
                       
                       
                        httppost.setRequestEntity(new FileRequestEntity(file));
                        HttpClient client = new HttpClient();
                        client.getHttpConnectionManager().getParams().setConnectionTimeout(
                                        5000);
                       
                        client.getState().setCredentials(
                                        new AuthScope(host, port,
                                                        AuthScope.ANY_REALM),
                                        new UsernamePasswordCredentials(user, password));
                        httppost.setDoAuthentication(true);
                       
                       
                        int status = client.executeMethod(httppost);
                       
                        if (status == HttpStatus.SC_OK) {
                                System.out.println("HttpStatus:"
                                        + httppost.getResponseBodyAsString());
                               
                        } else {
                       
                                System.out.println("Unexpected failure: " + httppost.getStatusLine().toString());
                       
                        }
               
                } finally {
                        if (httppost != null) {
                                httppost.releaseConnection();
                        }
                }

        }

</code snippet>

ANEESHCH wrote:

Hi,
I have a requirement for sending a huge stream, created by uploading a 2GB file, using fileupload to a remote file share using Http. When I'm tried with java.net.HttpURLConnection, it was failing by throwing OutOfmemory error.
1) Can I make use of HttpClient in this scenario?
2)What's the max file size which can be send using HttpClient?
3)Is it request streaming which I have to use for this purpose.

My application is running on, Websphere Application Server ND 5.1. and the current max heap size of eah JVM is 768M.


Any help is appreciated..

Thanks in advance

-A

[olegk]

On Sun, 2008-09-21 at 14:11 -0700, ANEESHCH wrote:

> Hi,
> I tried the following options so far.
> 1) Based on the unbuffered post, used
> httppost.setRequestEntity(new InputStreamRequestEntity(
>         new FileInputStream(file), file.length()));
>
> It gave me the error, org.apache.commons.httpclient.ProtocolException:
> Unbuffered entity enclosing request can not be repeated. in the system error
> file.
>
> 2) Then I tried using FileRequestEntity, to make sure that content can be
> recreated, as well as for marking the entity as repeatable
> No error was reported any where, but the file was not uploaded to the remote
> server.
> Httpstatus code 404 was returned by the excecutemethod();
>

Status code 404 means the server has not found anything matching the
Request-URI.

This problem has nothing to do with HttpClient

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[ANEESHCH]

Hi,
When I try the url, which I pass to the constructor of PostMethod from a browser I'm not getting 404. Also the same url was working with MultipartRequestEntity. When I was in the intranet, I was able to upload files upto 63Mb, with this method. But when I tried, a 240MB I started getting Sockettimeout exception, as well websphere reported some threads are hung. The same issue happened when I tried with a 1K file through VPN.
As a note, even though the code snippet which I posted, was hard coding the file, actually the file getting uploaded using struts 1.1 file upload, and the input stream from the upload form is passsed instead of the file.
You thoughts please..
-A

olegk wrote:

On Sun, 2008-09-21 at 14:11 -0700, ANEESHCH wrote:
> Hi,
> I tried the following options so far.
> 1) Based on the unbuffered post, used
> httppost.setRequestEntity(new InputStreamRequestEntity(
>         new FileInputStream(file), file.length()));
>
> It gave me the error, org.apache.commons.httpclient.ProtocolException:
> Unbuffered entity enclosing request can not be repeated. in the system error
> file.
>
> 2) Then I tried using FileRequestEntity, to make sure that content can be
> recreated, as well as for marking the entity as repeatable
> No error was reported any where, but the file was not uploaded to the remote
> server.
> Httpstatus code 404 was returned by the excecutemethod();
>

Status code 404 means the server has not found anything matching the
Request-URI.

This problem has nothing to do with HttpClient

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: [e-mail]
For additional commands, e-mail: [e-mail]