API: Downloading attachments >100MB throws com.atlassian.httpclient.api.ResponseTooLargeException

[Adam Głębowicz]

Hello,

I'm using the official JIRA API client:
jira-rest-java-client-api-5.2.6

I can download attachments from JIRA no problem, BUT if the attachment size is over 100 MB I get an Exception: com.atlassian.httpclient.api.ResponseTooLargeException: Entity content is too long; larger than 104857600 bytes

Is this a client-side or server-side issue?
Where can I change configuration to allow bigger responses than 100 MB?

Regards,
Adam

[Radek Dostál · Accepted solution]

Should be server-side. At least I think.

If you are a system administrator you can change the maximum allowed attachment size at System>Attachments (/secure/admin/ViewAttachmentSettings.jspa). If it's set to 100MB then that's it.

Technically there isn't really a "hardcap" for max size, but of course generally it is not recommended to be too high because then people start using it as storage and your backup sizes (assuming they include attachments) could grow badly.

[Adam Głębowicz]

Hello,

The maximum allowed attachment size has always been 800 MB on that JIRA System -> Attachments setting page.
We can upload and download (from the GUI) attachments as big as that.
But we get 100 MB limit issue on the API requests.

[Radek Dostál]

Yeah it hit me later that it's from the rest client probably and not from Jira (assuming it doesn't "use" Jira settings).

Digging into maven jar repositories there is a hardcoded variable in HttpClientOptions

private long maxEntitySize = 104857600L;

And there is a public setter as well

public void setMaxEntitySize(long maxEntitySize) {
    this.maxEntitySize = maxEntitySize;
}

I don't really use this client myself, can you show how you create an instance of the client / what you do before you use it? Then I'm curious if there is a way to set the max entity to a different value if we can get to the HttpClientOptions object. So an actual example how you initialize that client would help me a bit navigating around the jars (the HttpClientOptions is in a pom dependency of that client thing so it's a bit mixed).

I'm not really finding a good documentation for this thing. Lots of different versions, outdated stuff. If you could show what you add to pom and how you use it that'd help to play around with it.

[Adam Głębowicz]

Hello,

What's the goal here:
I iterate the JIRA IDs I need to migrate and all their Attachments.
I get an inputstream from each attachment and transfer it afterwards to Artifactory as a new attachment (we're moving the attachments).
The part that fails on JIRA side with the limit is highlighted in bold below.

I reference:
jira-rest-java-client-core-5.2.6.jar
jira-rest-java-client-api-5.2.6.jar
and all the compile dependencies for both, including the http libraries you mention.

import com.atlassian.jira.rest.client.api.JiraRestClient;
import com.atlassian.jira.rest.client.api.domain.Attachment;

JiraRestClient jiraClient = AsynchronousJiraRestClientFactory().createWithBasicHttpAuthentication(getJiraUri(JIRA_URL), JIRA_USER, JIRA_PASS);

Iterable<Attachment> jiraItemAttachments = jiraClient.getIssueClient().getIssue(JIRA_ID).claim().getAttachments();

for (Attachment attachment : jiraItemAttachments) {

InputStream attachmentInputStream = null;

URI attachmentUri = null;

       try {

              attachmentUri = attachment.getContentUri();

            attachmentInputStream = jiraClient.getIssueClient().getAttachment(attachmentUri).claim();

       } catch (Exception ex) {

             System.out.println("Error getting file " + attachment.getFilename() + " / " + ex);

       }

       (…)

}

If there's a good alternate method of doing the same thing with other API libraries that doesn't face this limit issue, I'd also be happy to use it.

Regards,
Adam

[Radek Dostál]

Sooo.. the good news is I have a sample maven project running with those dependencies, reused your example, and then was able to find a way to modify the HttpClientOptions which are used to create that HttpClient.

The problem is, it's got to such a point where I'm beginning to question if I'm sane for doing it.

Long story short, I don't see any way to touch those options in any way without "copying"(extending) the classes used in the process. Though, some of them are private inner classes and so that is where the fun begins with modifying a bit here and there.

All things considered, it probably would work, 

Of course, doing it this way I ended up with a lot of classpath dependencies so I had to add a few extra in pom to resolve those:

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.4.16</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.12</version>
    <scope>runtime</scope> <!-- I originally had provided, since I worked on a jira plugin, but if I changed this to runtime then it would work as a "local" java project -->
</dependency>
<dependency>
    <groupId>io.atlassian.fugue</groupId>
    <artifactId>fugue</artifactId>
    <version>5.0.0</version>
    <scope>runtime</scope> <!-- apparently needed by something in there -->
</dependency>
<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.5</version>
    <scope>compile</scope> <!-- needed this to save files to see if the thing works-->
</dependency>

Then I made a test class which tries to do what you're doing, and as you see it contains a LOT of extra noise and stuff in it:

package com.example.you.dont.need.to.know;

import com.atlassian.crowd.embedded.event.NoOpEventPublisher;
import com.atlassian.event.api.EventPublisher;
import com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient;
import com.atlassian.httpclient.apache.httpcomponents.DefaultHttpClientFactory;
import com.atlassian.httpclient.api.HttpClient;
import com.atlassian.httpclient.api.factory.HttpClientOptions;
import com.atlassian.jira.rest.client.api.AuthenticationHandler;
import com.atlassian.jira.rest.client.api.JiraRestClient;
import com.atlassian.jira.rest.client.api.domain.Attachment;
import com.atlassian.jira.rest.client.auth.BasicHttpAuthenticationHandler;
import com.atlassian.jira.rest.client.internal.async.*;
import com.atlassian.sal.api.ApplicationProperties;
import com.atlassian.sal.api.UrlMode;
import com.atlassian.sal.api.executor.ThreadLocalContextManager;
import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.annotation.Nonnull;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.Date;
import java.util.Optional;
import java.util.Properties;

public class Test {

    public static void main(String[] params) throws URISyntaxException, IOException {
        JiraRestClient jiraRestClient = new CustomAsynchronousJiraRestClientFactory().createWithBasicHttpAuthentication(new URI("https://baseurl.com"), "username", "password");

        Iterable<Attachment> attachments = jiraRestClient.getIssueClient().getIssue("ISSUEKEY-111").claim().getAttachments();
        for (Attachment attachment : attachments) {
            URI contentUri = attachment.getContentUri();
            InputStream inputStream = jiraRestClient.getIssueClient().getAttachment(contentUri).claim();
            // do something with inputstream
            File targetFile = new File("src/main/resources/" + attachment.getFilename());
            FileUtils.copyInputStreamToFile(inputStream, targetFile);
        }

    }

    static class CustomAsynchronousJiraRestClientFactory extends AsynchronousJiraRestClientFactory {
        public CustomAsynchronousJiraRestClientFactory() {
            super();
        }

        @Override
        public JiraRestClient createWithBasicHttpAuthentication(URI serverUri, String username, String password) {
            return this.create(serverUri, (AuthenticationHandler)(new BasicHttpAuthenticationHandler(username, password)));
        }

        @Override
        public JiraRestClient create(URI serverUri, AuthenticationHandler authenticationHandler) {
            DisposableHttpClient httpClient = (new CustomAsynchronousHttpClientFactory()).createClient(serverUri, authenticationHandler);
            return new AsynchronousJiraRestClient(serverUri, httpClient);
        }

    }

    static class CustomAsynchronousHttpClientFactory extends AsynchronousHttpClientFactory {
        public CustomAsynchronousHttpClientFactory() {
            super();
        }

        @Override
        public DisposableHttpClient createClient(URI serverUri, AuthenticationHandler authenticationHandler) {
            HttpClientOptions options = new HttpClientOptions();
            options.setMaxEntitySize(838860800L); //800MB
            final DefaultHttpClientFactory defaultHttpClientFactory = new DefaultHttpClientFactory(new CustomNoOpEventPublisher(), new CustomRestClientApplicationProperties(serverUri), new ThreadLocalContextManager() {
                public Object getThreadLocalContext() {
                    return null;
                }

                public void setThreadLocalContext(Object context) {
                }

                public void clearThreadLocalContext() {
                }
            });
            final HttpClient httpClient = defaultHttpClientFactory.create(options);
            return new AtlassianHttpClientDecorator(httpClient, authenticationHandler) {
                public void destroy() throws Exception {
                    defaultHttpClientFactory.dispose(httpClient);
                }
            };
        }

        private class CustomNoOpEventPublisher implements EventPublisher {
            public CustomNoOpEventPublisher() {
            }

            public void publish(Object event) {
            }

            public void register(Object listener) {
            }

            public void unregister(Object listener) {
            }

            public void unregisterAll() {
            }
        }

        private class CustomRestClientApplicationProperties implements ApplicationProperties {

            private final String baseUrl;

            private CustomRestClientApplicationProperties(URI jiraURI) {
                this.baseUrl = jiraURI.getPath();
            }

            public String getBaseUrl() {
                return this.baseUrl;
            }

            @Nonnull
            public String getBaseUrl(UrlMode urlMode) {
                return this.baseUrl;
            }

            @Nonnull
            public String getDisplayName() {
                return "Atlassian JIRA Rest Java Client";
            }

            @Nonnull
            public String getPlatformId() {
                return "jira";
            }

            @Nonnull
            public String getVersion() {
                // changed because private static classes etc. so much pain
                return new CustomAsynchronousHttpClientFactory.CustomMavenUtils().getVersion("com.atlassian.jira", "jira-rest-java-client-core");
            }

            @Nonnull
            public Date getBuildDate() {
                throw new UnsupportedOperationException();
            }

            @Nonnull
            public String getBuildNumber() {
                return String.valueOf(0);
            }

            public File getHomeDirectory() {
                return new File(".");
            }

            public String getPropertyValue(String s) {
                throw new UnsupportedOperationException("Not implemented");
            }

            @Nonnull
            public String getApplicationFileEncoding() {
                return StandardCharsets.UTF_8.name();
            }

            @Nonnull
            public Optional<Path> getLocalHomeDirectory() {
                return Optional.of(this.getHomeDirectory().toPath());
            }

            @Nonnull
            public Optional<Path> getSharedHomeDirectory() {
                return this.getLocalHomeDirectory();
            }
        }

        private class CustomMavenUtils {
            private final Logger logger = LoggerFactory.getLogger(CustomMavenUtils.class);
            private static final String UNKNOWN_VERSION = "unknown";

            private CustomMavenUtils() {
            }

            String getVersion(String groupId, String artifactId) {
                Properties props = new Properties();
                InputStream resourceAsStream = null;

                String var5;
                try {
                    resourceAsStream = CustomMavenUtils.class.getResourceAsStream(String.format("/META-INF/maven/%s/%s/pom.properties", groupId, artifactId));
                    props.load(resourceAsStream);
                    String var4 = props.getProperty("version", "unknown");
                    return var4;
                } catch (Exception var15) {
                    logger.debug("Could not find version for maven artifact {}:{}", groupId, artifactId);
                    logger.debug("Got the following exception", var15);
                    var5 = "unknown";
                } finally {
                    if (resourceAsStream != null) {
                        try {
                            resourceAsStream.close();
                        } catch (IOException var14) {
                        }
                    }

                }

                return var5;
            }
        }

    }

}

Then I ran that main as an "application" from my IDE.

I did end up downloading 6 test attachments I had on that test issue, though none of them above 100MB.

SO I guess if you really wanted it that much, this might be a workaround. BUT, this is seriously and sincerely bad and you will probably end up with dependency hell just as I did to get it to work in your project.

Could it be done better/prettier, probably. I could have made extra classes and not try to fit everything to a single "Test" class by creating so many inner classes in it. But I really was just curious if I would be able to dig deep enough to get it to run first.

Again though, should you do it, I would argue no. This is mad.

Personally, I tend to use https://hc.apache.org/httpcomponents-client-4.5.x/current/httpclient/apidocs/org/apache/http/impl/client/CloseableHttpClient.html in my projects. Sure you would need to code more, but it would be a much better way. Seems the Atlassian's REST client is not really too popular and that maxEntitySize is kind of hardcoded in it with no proper way of changing it without gutting it like this. Not worth it.

[Adam Głębowicz]

Hello,

Do you have any samples of using CloseableHttpClient in Your implementation to get  JIRA attachments?
When I tried alternatives to what I had, I remember trying this, but got stuck somewhere on authentication. I could download attachment metadata, but not the actual attachment contents. Instead of attachment contents I got a JIRA auth page HTML, because I was missing something to get authentication to properly work for this.

Regards,
Adam

[Radek Dostál]

Well, now we're kind of off the scope though. I don't think I can speak much about that since I never really had to write a service to download attachments.

Working primarily around API endpoints and stuff, this would be a simple way of doing it:

import org.apache.http.HttpEntity
import org.apache.http.HttpHeaders
import org.apache.http.client.methods.CloseableHttpResponse
import org.apache.http.client.methods.HttpGet
import org.apache.http.impl.client.CloseableHttpClient
import org.apache.http.impl.client.HttpClientBuilder
import org.apache.http.util.EntityUtils

CloseableHttpClient closeableHttpClient = HttpClientBuilder.create().build();
HttpGet httpGet = new HttpGet("https://...........");

// Add authorization cookie, PAT as "Bearer <value>" or "Basic <base64 encoded 'username:password'>
httpGet.setHeader(HttpHeaders.AUTHORIZATION, "Bearer XXX");

CloseableHttpResponse closeableHttpResponse = closeableHttpClient.execute(httpGet);

int statusCode = closeableHttpResponse.getStatusLine().getStatusCode();
HttpEntity entity = closeableHttpResponse.getEntity();

InputStream inputStream = entity != null ? entity.getContent() : null
//do something with inputStream; entity can be null if you get 404 or something so safer to handle that / log it, etc.


closeableHttpResponse.close() // always close the response when you no longer need it, otherwise it will hang around and consume slots in the client
closeableHttpClient.close() // always close the client when you no longer need it; not a big deal if there are no open connections lying around but anyway

Then again, I do not know IOO% you won't have problems with the file size, other than that I don't see there being any apparent limit. I also am not sure of the most appropriate way of handling the file, never had to, but found this on https://stackoverflow.com/questions/10960409/how-do-i-save-a-file-downloaded-with-httpclient-into-a-specific-folder

There are though plenty resources out on the internet around the apache httpclient so you have much better chances of finding stuff about it, I only know bare minimum for my own use cases.

[Adam Głębowicz]

Ok, great, thanks for the help :)

Regards,

Adam