Atlassian JIRA/Confluence in Azure slow and Marketplace unreachable

[Normann P_ Nielsen _Netic_]

I have recently migrated 2 customers (independant of each other) to linux machines in Azure.

On both, responsetimes got significant worse and og alle installation, marketplace cant be reached 50% of the time...

Regarding responsetime, the TTFB on files; espeicilly batch.js are sometimes extremely high.

I am working with the customers networkteam regarding the issues, but no real results so far.

But - I was wandering if anybody have similar experiences and/or some solutions. the one customer is new, so the its almost clean installations, the other customer has a large Jira and Confluence.

PS: Yes, I am aware of https://confluence.atlassian.com/jirakb/troubleshoot-performance-issues-in-jira-server-336169888.html etc...

[Kenneth Wannebo]

Hi @Normann P_ Nielsen _Netic_ 

Did you find a solution here?

We're having the exact same problems.

[Normann P_ Nielsen _Netic_]

Hi Kenneth,

NEVER !!

My customers "Azure expert" spent countless hours testing, setting up stuff and so on and so on....

Atlassian and Microsoft in generel just blamed each other for a few months on several tickets, and at some point I myself slipped out of the endless information stream, I for sure gave up....

I still have another customer in Azure an see the issue sometimes, and I just retry ... until it works... 

[Kenneth Wannebo]

Dissapointing.

We manage about a dozen jira and confluence instances deployed in Azure on both linux and windows VMs and different azure vm types.

All are experiencing the issues you are describing, though some worse than others. The problem does not seem to be deterministic.

[Andy Heinzer]

Hi Normann,

Sorry to hear about these problems.  It is very possible that Jira's slow performance and the unreachable marketplace could be related here.  I have seen problems like this could be caused by a number of different sources.  For example, if the base URL Jira is set to is unreachable by the Jira application (say due to an outbound proxy, or network configuration) it can cause Jira to be unable to connect to the marketplace site.

Please see our document on Problems Connecting to the Atlassian Marketplace. It explains in more depth the need for the application to be able to reach that site and common networking configurations that might be need to be adjusted to make this work. 

The other aspect I would be interested to learn more about would be how much JVM heap is being allocated for Jira to use.  Some of the previous versions of Jira server would only allocate 768m for the Xmx (max memory).  More recent versions are defaulting to 2048m, which is usually enough for most starting instances.  But of course as you get to larger scale, that memory usually needs to grow as well to keep up performance.  These are values a system admin can adjust when following steps in Increasing Jira application memory.

And of course if these answers do not help here, it probably would help to create a support case for these customers.  You can do that if you happen to know the SEN number on their account.  Or perhaps suggest that they create these so our support team can take a closer look at the support zip in order to better understand the environment and identify any problems that might cause this poor performance here.

Regards,

Andy

[Normann P_ Nielsen _Netic_]

Hi Andy,

Appreciated, - and +10 years of Jira here. As I said I have 2 independant customers facing both issues.

For one, the slow Jira could be related to the type of "B4MS" Azure VM, istead of "DS". We are trying to change that :-)

For the marketplace issue, one customer has been looking into the traffic from JIRA outwards, and states ( not my words).. :

"It seem that Jira reaches out for many things and its not possible to limit it tot the Azure, aws, confluence,..... ip adresses that are targets..

Last tested FW rule that was insuficcient was "allow for":

eg. a lot of different IPs ...

So now there is open for "any"..."

Quote end.

That was his findings, and I think that the problem is that firewalls normally does not do "domain based" firewalling, but is on IP level.

PS: Im not a network guy.....

BR,

Normann

[Andy Heinzer]

Hi Normann,

I totally understand.  I have worked with a few different firewall solutions and your teammates assessment is accurate to my own experience.  The ability to restrict web traffic to domains isn't always available.  Many of the physical firewalls will only accept IP address/range inputs.

We do have a list of domains and IP addresses that are needed to be accessible for our Jira Cloud offerings over in Atlassian cloud IP ranges and domains.  I understand that this isn't completely applicable in your case of using either Jira Server or Data Center, however when I did a bit of research on the complete IP address list we have in https://ip-ranges.atlassian.com/ I found that at least the main marketplace.atlassian.com address was included in one of those ranges over in

            "network": "13.52.5.0",
            "mask_len": 25,
            "cidr": "13.52.5.0/25",
            "mask": "255.255.255.128"

But this did not include a few other domains I checked such as maven.atlassian.com or marketplace-cdn.atlassian.com

We have tried to help document how we expect administrators to try to manage situations like this over in Configuring Marketplace connectivity.

Since IP addresses for a given domain can change over time, Atlassian recommends that you configure your firewall policy by hostname rather than by numeric IP address. If you do need to configure rules by IP address, we suggest that you use a network analysis tool to investigate outgoing connections made by UPM. And keep in mind that you may need to update the rules on occasion, in this case.

If that isn't feasible, then you might test out Switching UPM to offline mode at least temporarily.  Yes it would prevent your Jira site from connecting to Marketplace, but I'd be interested to see if this helps with other other performances aspects here.  If it does not, then perhaps the cause is something else.

Again, I think it might be best to create a support zip and create a support ticket over in https://support.atlassian.com/contact provided that you have the SEN (support entitlement number).  I'm sure our support team could better analyze any performance issues from the logs and perhaps they can offer some more concise troubleshooting based on your own environment.

Cheers,

Andy

[Normann P_ Nielsen _Netic_]

Well, opening for "any" actually did not solve it. So both customers still have some problems... no clue why..

I know about the disablement og UPM - thanks :-)

[Andy Heinzer]

@Normann P_ Nielsen _Netic_ Let's create a support case for each of these customers.  Do you know the SEN for each of these licenses?  If you do you could create this on behalf of each of those users in https://support.atlassian.com/contact

If you do not, then perhaps encourage these users to create the request themselves.  I could do this on your behalf as well, if I knew the SEN or email address of one of the billing or technical contacts on the license.

I believe we need to look at the logs to better understand the environment these Jira installations are using right now.

Thanks

Andy