Scene number?

[parkerc]

Hi

I’ve been investing a lot of time recently trying to digitise everything I can, and my current adventure is with all my paper documents.

My tool of choice is Paperless-ngx, which so far is an amazing open source solution, and highly recommended if you’re interest in such a thing…

Such a move to Paperless would not be complete without some form of integration into Vera / openLuup, and thankfully it has a nice Rest API (https://paperless-ngx.readthedocs.io/en/latest/api.html) I can use. So far I’ve been able to make use the GET requests, to provide document counts etc., but it’s the mutipart/form data piece where I’m struggling..

The Curl command to upload a document, is as follows

curl -H "Authorization: Basic Y2hyaXM62tgbsgjunotmeY2hyaXNob3N0aW5n" -F "title=Companies House File 10" -F "correspondent=12" -F "document=@/mnt/nas/10.pdf" http://192.168.102.134:8777/api/documents/post_document/

But, I’d like to know how to do this with Lua code, more just to help me in my Lua learning curve, and what I thought would be reasonably straight forward thing to do, has turned out to be quite the opposite Sending a multipart form data submission via http.request, is not as straight forward as I hoped. If anyone has any time to help, I’d appreciate some guidance on what I’m missing with the following, as based on the feedback, it looks like it not seeing the file (which I think I’ve confirmed by trying to sending both the curl and Lua commands to httbin.org ) …

local http = require("socket.http")
local ltn12 = require("ltn12")
local mime = require("mime")
local lfs = require("lfs")

local username = "username"
local password = "password"

local httpendpoint = 'http://httpbin.org/post'
local filepath = "/mnt/nas/10.pdf"
local file = io.open(filepath, "rb")
local contents = file:read( "*a" )

-- https://stackoverflow.com/questions/3508338/what-is-the-boundary-in-multipart-form-data

local boundary = "-----BoundaryePkpFF7tjBAqx29L"
local send = "--"..boundary..
            "\r\nContent-Disposition: form-data; title='Companies House File'\r\n"
			.."----"..boundary..
			"\r\nContent-Disposition: form-data; document="..filepath..
            "\r\nContent-type: application/pdf".."\r\n"
			..contents.."\r\n"
			.."------"..boundary.."--\r\n"

-- Execute request (returns response body, response code, response header)

local resp = {}
local body, code, headers, status = http.request {
    url = httpendpoint,
    method = 'POST',
    headers = {
		["Authorization"] = "Basic " .. (mime.b64(username ..":" .. password)),
		["Content-Length"] = lfs.attributes(filepath, 'size'),
		['Content-Type'] = "multipart/form-data; boundary="..boundary,
    },
   -- source = ltn12.source.file( io.open(filepath,"rb") ),
   source = ltn12.source.file( io.open(send) ),
    sink = ltn12.sink.table(resp)
}

print(body, code, headers, status)
print(table.concat(resp))

if headers then 
    for k,v in pairs(headers) do 
        print(k,v) 
    end
end

[parkerc]

--

To provide some additional information, the following Curl command sent to httpbin.org/post..

local url_open = 'curl -u "username:password" -F "title=Companies House File" -F "correspondent=12" -F "document=@/mnt/nas/10.pdf" http://httpbin.org/post'
local command = io.popen(url_open)
local result = command:read("*a")
print(result)

Returns the following, which looks good…

{
"args": {}, 
"data": "", 
"files": {
"document": "data:application/octet-stream;base64,JVBERi0xLjMNJeLjz9MNCjIyNyAwIG9iag08PCANL0xpbmVhcml6ZWQgMSANL08gMjMxIA0vSCBbIDIzNTAgODc5IF<REDACTED>T5dDT4+DXN0YXJ0eHJlZg0xNzMNJSVFT0YN"
}, 
"form": {
"correspondent": "12", 
"title": "Companies House File"
}, 
"headers": {
"Accept": "*/*", 
"Authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", 
"Content-Length": "66501", 
"Content-Type": "multipart/form-data; boundary=------------------------1a4c10e40f1ccf8e", 
"Host": "httpbin.org", 
"User-Agent": "curl/7.38.0", 
"X-Amzn-Trace-Id": "Root=1-6360dbdb-6bea68142a74a1762e74ca04"
}, 
"json": null, 
"origin": "82.30.93.28", 
"url": "http://httpbin.org/post"
}

Then using the following Lua code to try and achieve the same thing..

local http = require("socket.http")
local ltn12 = require("ltn12")
local mime = require("mime")
local lfs = require("lfs")

local username = "username"
local password = "password"

local httpendpoint = 'http://httpbin.org/post'
local filepath = "/mnt/nas/10.pdf"
local file = io.open(filepath, "rb")
local contents = file:read( "*a" )

-- https://stackoverflow.com/questions/3508338/what-is-the-boundary-in-multipart-form-data

local boundary = "-----BoundaryePkpFF7tjBAqx29L"
local send = "--"..boundary..
            "\r\nContent-Disposition: form-data; title='Companies House File'\r\n"
			.."----"..boundary..
			"\r\nContent-Disposition: form-data; document="..filepath..
            "\r\nContent-type: application/pdf".."\r\n"
			..contents.."\r\n"
			.."------"..boundary.."--\r\n"

-- Execute request (returns response body, response code, response header)

local resp = {}
local body, code, headers, status = http.request {
    url = httpendpoint,
    method = 'POST',
    headers = {
		["Authorization"] = "Basic " .. (mime.b64(username ..":" .. password)),
		["Content-Length"] = lfs.attributes(filepath, 'size'),
		['Content-Type'] = "multipart/form-data; boundary="..boundary,
    },
   -- source = ltn12.source.file( io.open(filepath,"rb") ),
   source = ltn12.source.file( io.open(send) ),
    sink = ltn12.sink.table(resp)
}

print(body, code, headers, status)
print(table.concat(resp))

if headers then 
    for k,v in pairs(headers) do 
        print(k,v) 
    end
end

I get this returned..

nil     -------BoundaryePkpFF7tjBAqx29L 
Content-Disposition: form-data; title='Companies House File' 
---------BoundaryePkpFF7tjBAqx29L 
Content-Disposition: form-data; document=/mnt/nas/10.pdf 
Content-type: application/pdf 
%PDF-1.3 %âãÏÓ 
227 0 obj << /Linearized 1 /O 231 /H [ 2350 879 ] /L 66081 /E 19639 /N 4 /T 61422 >> endobj xref 227 57 0000000016 00000 n 
0000001509 00000 n 
0000001585 00000 n 
0000001726 00000 n 
0000003229 00000 n 
0000003404 00000 n 
0000003589 00000 n 
0000003808 00000 n 
0000003956 00000 n 
0000004161 00000 n 
0000004309 00000 n 
0000004515 00000 n 
0000004663 00000 n 
0000004843 00000 n 
0000005015 00000 n 
0000005218 00000 n 
0000005366 00000 n 
0000005570 00000 n 
0000005718 00000 n 
0000006003 00000 n 
0000006289 00000 n 
0000006456 00000 n 
0000006549 00000 n 
0000006804 00000 n 
0000007035 00000 n 
0000007265 00000 n 
0000007461 00000 n 
0000007655 00000 n 
0000007851 00000 n 
0000008046 00000 n 
0000008253 00000 n 
0000008400 00000 n 
0000008622 00000 n 
0000008770 00000 n 
0000008982 00000 n 
0000009130 00000 n 
0000009353 00000 n 
0000009501 00000 n 
0000009720 00000 n 
0000009866 00000 n 
0000010075 00000 n 
0000010223 00000 n 
0000010948 00000 n 
0000011372 00000 n 
0000011394 00000 n 
0000011503 00000 n 
0000012853 00000 n 
0000013021 00000 n 
0000013133 00000 n 
0000014623 00000 n 
0000014730 00000 n 
0000016927 00000 n 
0000016950 00000 n 
0000019315 00000 n 
0000019394 00000 n 
0000002350 00000 n 
0000003207 00000 n 
trailer << /Size 284 /Info 226 0 R /Encrypt 229 0 R /Root 228 0 R /Prev 61411 /ID[] >> startxref 0 %%EOF 228 0 obj << /Type /Catalog /Pages 225 0 R /AcroForm 230 0 R >> endobj 229 0 obj << /Filter /Standard /V 1 /R 2 /O (ÞŒŠþ0á6xƒV\nX¹#'ügã;Èè_Žu®ãªæu_-) /U (Ôeè³~„ˆ¸ÚC»ÒV!Øm: No such file or directory     

It’s clear I’m not getting the content sent in the right format/structure, but I cannot find any guidance that clearly explains how to construct a multi-part form submissions via Lua code..

Hoping someone can help…

[parkerc]

Managed to get there in the end, posting here to help others..

local http = require("socket.http")
local ltn12 = require("ltn12")
local mime = require("mime")
local lfs = require("lfs")

local username = "username,"
local password = "password"

http.TIMEOUT = 5

local function upload_file ( url, filename, title, correspondent )
    local fileHandle = io.open( filename,"rb")
    local fileContent = fileHandle:read( "*a" )
    fileHandle:close()

    local boundary = 'abcd'
	local cdfd = 'Content-Disposition: form-data; '

    local header_b = cdfd.. 'name="document"; filename="' ..filename.. '"\r\nContent-Type: application/pdf'
    local header_c = cdfd.. 'name="title"\r\n\r\n'..title
    local header_d = cdfd.. 'name="correspondent"\r\n\r\n'..correspondent

    local MP_b = '--'..boundary..'\r\n'..header_b..'\r\n\r\n'..fileContent..'\r\n'
    local MP_c = '--'..boundary..'\r\n'..header_c..'\r\n'
    local MP_d = '--'..boundary..'\r\n'..header_d..'\r\n'

    local MPCombined = MP_b..MP_c..MP_d..'--'..boundary..'--\r\n'

    local   response_body = { }
    local   _, code = http.request {
            url = url ,
            method = "POST",
            headers = {    
						["Authorization"] = "Basic " .. (mime.b64(username ..":" .. password)),
						["Content-Length"] =  MPCombined:len(),
                        ['Content-Type'] = 'multipart/form-data; boundary=' .. boundary
                         },
            source = ltn12.source.string(MPCombined) ,
            sink = ltn12.sink.table(response_body),
                }
     return code, table.concat(response_body)
end

 -- local rc,content = upload_file ('http://httpbin.org/post', '/mnt/nas/10.pdf' )
 local rc,content = upload_file ('http://192.168.1.134:8777/api/documents/post_document/', '/mnt/nas/10.pdf', 'Companies House', '12')
 print(rc,content)