BitBucket

[Rafal Sojka]

Hi,

We have BitBucket 5.6.1 in my company and yesterday I found this exception in the logs:

2018-01-27 09:48:11,860 WARN [threadpool:thread-6] c.a.b.i.w.d.AsyncInvocationHistoryDao Failed to write webhook invocation data for 13:pr:opened to the database
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
Database:
 - name:Oracle
 - version:Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics, Real Application Testing and Unified Auditing options
 - minor version:1
 - major version:12
Driver:
 - name:Oracle JDBC driver
 - version:12.2.0.1.0

java.sql.SQLException: ORA-12899: value too large for column "BITBUCKETAPP"."AO_371AEF_HIST_INVOCATION"."REQUEST_URL" (actual: 520, maximum: 255)

 at com.att oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494)
 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446)
 at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054)
 at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623)
 at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
 at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
 at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
 at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59)
 at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:910)
 at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
 at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
 at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)
 at oracle.jdbc.driver.OraclePreparedStatement.executeLargeUpdate(OraclePreparedStatement.java:3865)
 at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3845)
 at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1061)
 at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
 at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
 at net.java.ao.db.OracleDatabaseProvider.executeInsertReturningKey(OracleDatabaseProvider.java:297)
 at net.java.ao.DatabaseProvider.insertReturningKey(DatabaseProvider.java:1861)
 at net.java.ao.EntityManager.create(EntityManager.java:366)
 at net.java.ao.EntityManager.create(EntityManager.java:399)
 at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:100)
 ... 19 common frames omitted

 Everything seems to work fine, users do not complain, I can commit and checkout changes but the exception appears all the time in the logs. Do you know what can be the reason ?

We have only one plugin named "Stash Webhook to Jenkins" but it has been disabled some time ago so it should not have impact right now.

Thank you for any help,
Rafi

[Rafal Sojka]

I found the possible reason of the issue.

One of our users created a very long encrypted URL having 540 chars for webhook.

How is that possible that you do not validate it before saving the webhook ?

Database can take only 255chars for the field.

We upgraded the current BitBucket in December, is that possible that some database upgrade scripts did not work as expected ? 

Regards
Rafal

[Rafal Sojka]

Could you please raise a bug for Atlassian Support ? Unfortunately I cannot do it on my own because receive an exception (no "Assign Issue" permission). Thank you.

[Nic Brough -Adaptavist-]

Go to https://support.atlassian.com/contact and raise it there.

[Steve Thornhill]

If you wish to raise a support ticket you will have to do this directly.

As to the url length, you could update it to use a tinyurl version to stick within the table size limits.

[Steve Thornhill]

This sort of error is normally due to data being truncated when writing to the table as the format is wrong, such as trying to insert the wrong data type into a table.

As the error is showing on one of the AO tables, and you mention the plugin is disabled - can you try removing the plugin (if it is no longer needed) and see if the error still occurs?

Additionally, tail the log file while you are performing some updates to see which operation is causing the error to appear.

[Rafal Sojka]

Thank you for response.

The plugin was removed but the error is still there, so it was not generated by this disabled plugin, unfortunately.

It seems that someone created a webhook/hook in a repository that causes the problem however in the logs I haven't found any data that could help me in identification of this repository. However there are such labels appearing in every newly generated stacktrace:

• 13:pr:opened or
  
• 13:pr:comment:added

Is there any way to identify what 13 means in this case, maybe I could execute some SQL queries in database to find information about it ? Maybe it is a project id ?

The error appears in the logs all the time for more 15 days, it seems two weeks ago someone has created something but it has not impacted BitBucket in general that is why I found this error accidentally by simply looking into the logs.

Please advise if you have any idea I could follow up. 

Thank you very much for your help and support,

Rafal

[Rafal Sojka]

Hi

Is there any procedure that can be applied for such issues ? If not, I think we should raise a Support ticket for it.

Thank you,
Rafal