How to import hundreds of .md files

[Kevin Gaspard]

Hello,

my company just had his own Confluence and we would like to migrate hundreds of .md pages from our actual wiki.

After a few tests I was able to add manually one of our page by pasting the markdown source code to generate a nice looking formatting. It works and it's fine, but we need to do it as fast and easily as we can, without waste of time for humans.

Our first guess would be to have access to a page where we can drag'n'drop the hundreds of page or a FTP/sFTP access to do the same thing. But I'm not sure this is possible.

What's the quickest way to do that please? Do you have a piece of advice for us?

Regards,

Kévin GASPARD

[Thomas Deiler]

Dear @Kevin Gaspard,

I do not know your format ".md", but I would give it a try with the REST API.

So long

Thomas

[Thomas Deiler]

The REST call

POST https://myconfluence.com/rest/content

The JSON payload:

{

 "type":"page",
 "title":"A new page",
 "space":{"key":"ABC"},
 "body":{"
       storage":{
            "value":"place your .md content per page here",
            "representation":"storage"
       }
    }
}

Per new page, one call. 

[Kevin Gaspard]

Thanks for that quick answer !

FYI, the .md is the file extension for Markdown: https://fileinfo.com/extension/md

We'll take a look at this, but if someone else have an other idea that would be welcome.

Regards,

Kévin GASPARD

[Thomas Deiler]

Dear @Kevin Gaspard,

that seems to be easy. If you have one .md file per page, than you can just loop with a script over all files and fire one request.

My estimation for that script would be 1-2 hours writing, including testing on a test environment, first. Then the import of 1000 pages will not take longer than a couple of minutes.

So long

Thomas

[Kevin Gaspard]

Again, thank you very much.

We will try this and keep you updated in this topic, maybe it can help someone else.

Regards,

Kévin GASPARD

[Kevin Gaspard]

I tried to make it works by trying to send a single request with curl, like this:

curl -u kevin.gaspard:mySuperPassword. -X POST -H 'Content-Type: application/json' -d'{"type":"page","title":"TEST REST API", "space":{"key":"TST"},"body":{"storage":{"value":"<p>Je suis un test de REST API</p>","representation": "storage"}}}' https://mycompany.atlassian.net/wiki/rest/api/content | python -mjson.tool

But it give me this output:

{
    "data": {
        "authorized": false,
        "errors": [],
        "successful": false,
        "valid": true
    },
    "message": "Could not create content with type page",
    "statusCode": 403
}

But I just copied and modified what I saw on this page: https://developer.atlassian.com/server/confluence/confluence-rest-api-examples/

So maybe this documentation is not up to date ? Any ideas about this ? What should be the good type for this request ?

Regards,
Kévin GASPARD

[Thomas Deiler]

Dear @Kevin Gaspard,

from the upper address "mycompany.atlassian.net" I guess that you are using Confluence Cloud, right? Your documentation link is for Server. Right Link.

So long

Thomas

[Thomas Deiler]

Dear @Kevin Gaspard,

for me following worked:

POST https://......atlassian.net/wiki/rest/api/content

JSON Payload:

{
 "ancestors": [
 {
 "id": "123456"
 }
 ],
 "body": {
 "storage": {
 "value": "This is some text.",
 "representation": "storage" 
 }
 },
 "space": {
 "key": "TST"
 },
 "status": "current",
 "title": "New Title",
 "type": "page"
 }

 The id has to be the id of the parent page. You can find this out, when showing the Page Information of a Confluence Page.

So long

Thomas

[Kevin Gaspard]

Thanks Thomas!

From your example I tried this:

curl --noproxy "*" --user kevin.gaspard@mycompany.com --header 'Content-Type: application/json' -x POST --data '{ "ancestors": [ { "id": "xxxxxx" } ], "body": { "storage": { "value": "This is some text.", "representation": "storage" } }, "space": { "key": "TST" }, "status": "current", "title": "Test", "type": "page" }' --url https://mycompany.atlassian.net/wiki/rest/api/content

This is the output:

 {"statusCode":403,"data":{"authorized":false,"valid":true,"errors":[],"successful":false},"message":"Could not create content with type page"}

I'm sure the credentials are perfectly fine, I tried them in my web browser, I tried a fake credentials and the output is different from this.

Also, my account is able to create a page.

The ID is the ID from the URL in the section where I want to create a new page.

[Thomas Deiler]

Dear @Kevin Gaspard,

the Authentication Header is missing, or?

 -H "Authorization:Basic YWrtaW36TE9HSXTlY2gx"

Best you use a REST Client Plugin for your browser and check if it works.

So long

Thomas

[Thomas Deiler]

curl -i -X POST \
 -H "Content-Type:application/json" \
 -H "Authorization:Basic YWrtaW36TE9HSXTlY2gx" \
 -d \
'{
 "ancestors": [
 {
 "id": "123456"
 }
 ],
 "body": {
 "storage": {
 "value": "This is some text.",
 "representation": "storage" 
 }
 },
 "space": {
 "key": "TEST"
 },
 "status": "current",
 "title": "New Title",
 "type": "page"
 }' \
 'https://___________.atlassian.net/wiki/rest/api/content'

[Kevin Gaspard]

I appreciate your help, thanks again Thomas!

I guess this is a key private and related to the Confluence of my company ?

-H "Authorization:Basic YWrtaW36TE9HSXTlY2gx"

I should ask the admin to give me our API key ?

Regards,
Kévin GASPARD

[Thomas Deiler]

It's possible, that user/pass will not work, but OAuth instead, you are right.

[Kevin Gaspard]

Hello !

Sorry but I come back because I am not sure about something, my last question was about this line in the code you pasted:

-H "Authorization:Basic YWrtaW36TE9HSXTlY2gx"

Can you confirm that we need to use a personal API Key, which is not the same as in your pasted code ? So our administrator have to give me our Rest API Key ?

I'm wondering this because if I try your code with the ID of one of our page, I get this error (short version):

«HTTP Status 401 - Basic Authentication Failure - Reason : com.atlassian.crowd.exception.AccountNotFoundException: Account with name &lt;&gt; could not be found»

Also, why the user/password authentication might not work ? It would be nice for me to make it works this way because I don't have to annoy our administrator about this API Key.

PS: Sorry if the answer is obvious and in your last post, English isn't my native language.

Thanks for your time.

[Thomas Deiler]

Dear @Kevin Gaspard,

the upper line is no API key, it's just your user/password combination in a hashed manner.

Of cause this one will not be valid for your system.

Do following: install a "rest client" for your browser (browser extension). There you can comfortable combine one request and then test it.

Some of them have also a curl export function.

So long

Thomas

[Stephen Deutsch]

Hi Kevin,

I stumbled across this question, and it seems like the method that has been described to you won't work so well on its own, as you would have to convert the markdown format to storage format first before importing it as the page code, otherwise you would just get a page with the text of the markdown instead of a nicely formatted document.

Luckily, you don't have to go through all that, because Atlassian has a marketplace app that makes importing pages into Confluence Cloud really easy. It's called the Confluence Universal Importer and could import all your markdown files in one go, and it's free.

[Michael Barbulescu]

unfortunately Confluence Universal Importer tells "Looks like you took a wrong turn",
so try this filter 

[Bill Bailey]

To help you out, we stumbled across this package for Python:

https://github.com/pushrodtechnology/PythonConfluenceAPI