Confluence migration from Server to Cloud is failing due to attachments load error

[PavelKarpenka]

Hello dear Atlassian Experts!

We are trying to migrate form Confluence Server 7.4.1 to our existing Confluence Cloud. 
Some spaces were migrated successfully except the biggest one (and most important).

We are getting this error in Migration Assistant UI:
```
Failed to read 4 attachment(s). Check the Confluence log file for more details.
```
The Confluence log ends with the following:
```
...
2020-06-29 13:46:28,455 INFO [Caesium-1-1] [service.stepexecutor.attachment.AttachmentMigrationExecutor] info Upload status for spaceKey: RD - com.atlassian.migration.agent.service.UploadState@4629b116[numOfAttachments=533,numOfUploadedAttachments=529,uploadedBytes=91542597,percentOfProgress=99,totalBytes=91605702]
-- stepConfig: RD | cloudId: 350a5993-b7fa-4a71-885b-e55f8f093db6 | planName: to-ourcloud | stepType: ATTACHMENT_UPLOAD | stepId: 374d108a-1109-4f83-8577-ed7422bcc7d7 | planId: 6bbe7cad-027b-4c3a-a51b-4625956a802e | cloudUrl: https://orcloud.atlassian.net | taskName: Migrate space RD | taskId: 4f0d903d-7df7-4f5d-bc8b-361fae3950c6
2020-06-29 13:46:28,498 INFO [Caesium-1-1] [service.stepexecutor.attachment.AttachmentMigrationExecutor] info Attachment count for spaceKey: RD, contentAttachmentCount: 533, migAttachmentCount: 529, unRetrievableMigAttachmentCount: 4
-- stepConfig: RD | cloudId: 350a5993-b7fa-4a71-885b-e55f8f093db6 | planName: to-ourcloud | stepType: ATTACHMENT_UPLOAD | stepId: 374d108a-1109-4f83-8577-ed7422bcc7d7 | planId: 6bbe7cad-027b-4c3a-a51b-4625956a802e | cloudUrl: https://ourcloud.atlassian.net | taskName: Migrate space RD | taskId: 4f0d903d-7df7-4f5d-bc8b-361fae3950c6
2020-06-29 13:46:28,500 ERROR [Caesium-1-1] [agent.service.stepexecutor.ProgressTracker] error Step failed, message: Failed to read 4 attachment(s). Check the Confluence log file for more details.
-- stepConfig: RD | cloudId: 350a5993-b7fa-4a71-885b-e55f8f093db6 | planName: to-ourcloud | stepType: ATTACHMENT_UPLOAD | stepId: 374d108a-1109-4f83-8577-ed7422bcc7d7 | planId: 6bbe7cad-027b-4c3a-a51b-4625956a802e | cloudUrl: https://ourcloud.atlassian.net | taskName: Migrate space RD | taskId: 4f0d903d-7df7-4f5d-bc8b-361fae3950c6
```

Above in the log I see 4 errors like this one:

```
2020-06-29 13:43:05,269 WARN [pool-34-thread-6] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
->[PluginReadWriteTx]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #2005992115)
-- planName: to-ourcloud | cloudId: 350a5993-b7fa-4a71-885b-e55f8f093db6 | planId: 6bbe7cad-027b-4c3a-a51b-4625956a802e | cloudUrl: https://ourcloud.atlassian.net | taskName: Migrate space RD | attachmentId: 328302 | taskId: 4f0d903d-7df7-4f5d-bc8b-361fae3950c6
2020-06-29 13:43:05,272 WARN [pool-34-thread-6] [service.stepexecutor.attachment.AttachmentMigrator] warn Failed to read attachment 328302 data. The error is unrecoverable. The attachment will be marked as broken and will not be retried.
-- planName: to-ourcloud | cloudId: 350a5993-b7fa-4a71-885b-e55f8f093db6 | planId: 6bbe7cad-027b-4c3a-a51b-4625956a802e | cloudUrl: https://ourcloud.atlassian.net | taskName: Migrate space RD | attachmentId: 328302 | taskId: 4f0d903d-7df7-4f5d-bc8b-361fae3950c6
com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentDataProvider$AttachmentDataRetrievalException: Could not get attachment 328302 data
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentDataProvider.lambda$getAttachmentData$0(AttachmentDataProvider.java:54)
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)
at com.atlassian.sal.spring.component.SpringHostContextAccessor.lambda$doInTransaction$0(SpringHostContextAccessor.java:70)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at com.atlassian.sal.spring.component.SpringHostContextAccessor.doInTransaction(SpringHostContextAccessor.java:68)
at com.atlassian.confluence.spring.transaction.interceptor.ConfluenceSpringHostContextAccessor.access$001(ConfluenceSpringHostContextAccessor.java:21)
at com.atlassian.confluence.spring.transaction.interceptor.ConfluenceSpringHostContextAccessor.lambda$doInTransaction$3(ConfluenceSpringHostContextAccessor.java:72)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:84)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:68)
at com.atlassian.confluence.spring.transaction.interceptor.ConfluenceSpringHostContextAccessor.doInTransaction(ConfluenceSpringHostContextAccessor.java:72)
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:18)
at jdk.internal.reflect.GeneratedMethodAccessor268.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at com.sun.proxy.$Proxy205.execute(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor268.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy2679.execute(Unknown Source)
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentDataProvider.getAttachmentData(AttachmentDataProvider.java:39)
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentMigrator.uploadImpl(AttachmentMigrator.java:72)
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentMigrator.lambda$upload$0(AttachmentMigrator.java:65)
at com.atlassian.migration.agent.logging.LoggingContextBuilder.execute(LoggingContextBuilder.java:127)
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentMigrator.upload(AttachmentMigrator.java:65)
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentMigrator.migrate(AttachmentMigrator.java:45)
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentsToMigrateConsumer.run(AttachmentsToMigrateConsumer.java:54)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
at com.atlassian.migration.agent.logging.LoggingContextAwareExecutorService.lambda$execute$3(LoggingContextAwareExecutorService.java:104)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.atlassian.confluence.pages.AttachmentDataNotFoundException: Problem while getting attachment stream (Attachment: Need Title.graphml v.1 (328302) naor) from file system
at com.atlassian.confluence.pages.persistence.dao.filesystem.FileSystemAttachmentDataDao.getDataForAttachment(FileSystemAttachmentDataDao.java:65)
at com.atlassian.confluence.pages.persistence.dao.filesystem.FileSystemAttachmentDataDao.getDataForAttachment(FileSystemAttachmentDataDao.java:54)
at com.atlassian.confluence.pages.persistence.dao.filesystem.FileSystemAttachmentDataDao.getDataForAttachment(FileSystemAttachmentDataDao.java:46)
at com.atlassian.confluence.pages.persistence.dao.hibernate.AbstractHibernateAttachmentDao.getAttachmentData(AbstractHibernateAttachmentDao.java:341)
at com.atlassian.confluence.pages.attachments.CachingAttachmentDao.getAttachmentData(CachingAttachmentDao.java:192)
at com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentDataProvider.lambda$getAttachmentData$0(AttachmentDataProvider.java:51)
... 47 more
Caused by: com.atlassian.confluence.pages.persistence.dao.filesystem.AttachmentDataFileSystemException: No such file for Attachment: Need Title.graphml v.1 (328302) naor. Were looking at /var/atlassian/application-data/confluence/attachments/ver003/21/241/491521/238/78/328238/328300/1
at com.atlassian.confluence.pages.persistence.dao.filesystem.HierarchicalMultiStreamAttachmentDataFileSystem.getAttachmentData(HierarchicalMultiStreamAttachmentDataFileSystem.java:336)
at com.atlassian.confluence.pages.persistence.dao.filesystem.FileSystemAttachmentDataDao.getDataForAttachment(FileSystemAttachmentDataDao.java:63)
... 52 more
2020-06-29 13:43:05,273 WARN [pool-34-thread-6] [agent.service.impl.DefaultAttachmentService] warn Failed attachment migration for attachment 328302 to cloud 350a5993-b7fa-4a71-885b-e55f8f093db6
-- planName: to-ourcloud | cloudId: 350a5993-b7fa-4a71-885b-e55f8f093db6 | planId: 6bbe7cad-027b-4c3a-a51b-4625956a802e | cloudUrl: https://ourcloud.atlassian.net | taskName: Migrate space RD | attachmentId: 328302 | taskId: 4f0d903d-7df7-4f5d-bc8b-361fae3950c6
```

Short error messages are:

```

No such file for Attachment: Need Title.graphml v.1 (328302) naor. Were looking at /var/atlassian/application-data/confluence/attachments/ver003/21/241/491521/238/78/328238/328300/1

No such file for Attachment: Need Title.graphml v.2 (328306) naor. Were looking at /var/atlassian/application-data/confluence/attachments/ver003/21/241/491521/238/78/328238/328300/2

No such file for Attachment: test1.graphml v.2 (328347) eytan. Were looking at /var/atlassian/application-data/confluence/attachments/ver003/21/241/491521/238/78/328238/328347/2

No such file for Attachment: test1.graphml v.1 (328349) eytan. Were looking at /var/atlassian/application-data/confluence/attachments/ver003/21/241/491521/238/78/328238/328347/1

```

I googled a little bit and found that the issue can be related to yWorks diagram. So I converted all my yWorks diagrams to simple images. No effect.
Then I even deleted articles that previously contained yWorks diagrams. The same error during migration.

Could you please help me to figure out what I'm doing wrong?

Thanks a lot!

[Christian Schopf]

Have the same issue with 5 attachments in one space.
Went in confluence for the logfile analyzer. there you find something like 
"Pages can not be copied or images not loaded on a page" with a line number from the atlassian-confluence.log

I created the support.zip
Extracted it.
looked into the atlassian-confluence.log via notepad++

There I found entries like these:

-- planName: Test Migration | cloudId: 5cdbb56b-5b72-45e6-87bc-3bbdf779503a | planId: 0895507e-565b-47ad-8971-e893789ace32 | cloudUrl: https://xxx.atlassian.net | taskName: Migrate space SPACEXXX| attachmentId: 23206442 | taskId: 8bd998ea-80d7-401b-a8a4-7d4a24affefe
com.atlassian.migration.agent.service.stepexecutor.attachment.AttachmentDataProvider$AttachmentDataRetrievalException: Could not get attachment 23206442 data. Attachment download path: https://XXX.ONPREMURL.TLD/download/attachments/23206440/image2019-4-15_15-55-0.png?version=1 please open and verify

 Searching for "DataProvider$AttachmentDataRetrievalException: Could not get attachment" found 5 entries.

Used the URLs from the log in the browser. e.g.
https://XXX.ONPREMURL.TLD/download/attachments/23206440/image2019-4-15_15-55-0.png?version=1

Showed an error about missing attachments with the possibility to go "back to attachments page"
There I looked for the attachment e.g. image2019-4-15_15-55-0.png
Deleted it.

Repeated it for the 5 attachments.
Voilá worked.

[vinci-mike]

Did you ever have luck here? We have the same issue. It also says "unRetrievableMigAttachmentCount" is 4.

We've run the tool to fix missing attachments (https://bitbucket.org/atlassianlabs/confluence-support-fix-missing-attachments/src/master/). There were 4 attachments it couldn't fix, but we removed those attachments from the page that had them, then deleted (and purged) that page. The missing attachments tool thinks everything's fine.

We've shutdown and restarted Confluence.

We've rebuilt the content index.

.. but we still get the same error. The Confluence log for the migration says "Added attachment" 278 times. Then it says UPLOAD_SKIPPED 278 times (I think it has already uploaded all attachments to the destination site). But then it errors out at the end with:

[service.stepexecutor.attachment.AttachmentMigrationExecutor] info Upload status for spaceKey: ~<personal space> - UploadState(numOfAttachments=378, numOfUploadedAttachments=0, uploadedBytes=0, percentOfProgress=0, totalBytes=170806383, numOfFailedAttachments=0)

[service.stepexecutor.attachment.AttachmentMigrationExecutor] info Attachment count for spaceKey: ~<personal space>, contentAttachmentCount: 378, migAttachmentCount: 374, unRetrievableMigAttachmentCount: 4
[agent.service.stepexecutor.ProgressTracker] error Step failed, message: Failed to read 4 attachment(s). Check the Confluence log file for more details.

I made a script to analyze the log output, and I've verified that all 378 attachment IDs are added to be uploaded, and then the same 378 IDs are mentioned as UPLOAD_SKIPPED. I can't find any special 4 attachments mentioned in the logs.

[vinci-mike]

Finally fixed it. It turned out that there was a deleted page in the personal space that we were trying to migrate, and that had references to attachments that didn't exist. The fix was to purge the trash in that space, and then the migration worked.