Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..

[parkerc]

I have a couple of calendar URLs I’m trying to call, and both work when i enter the URL into the browser, however when I try to call them via Lua and http(s).request, only one of them works ? Please could someone help me understand why, and what needs to be done differently to make the other one work ?

local socket = require 'socket'
local http = require "socket.http"
local ssl = require 'ssl'
local https = require 'ssl.https'

local calendarUria = "https://ics.fixtur.es/v2/ipswich-town.ics"
local calendarUrib = "https://calendar.google.com/calendar/ical/en-gb.uk%23holiday%40group.v.calendar.google.com/public/basic.ics"

print("Fixtur.es Football (URL A) returns the following?")
local responseBodya, responseCodea, responseHeadersa = https.request(calendarUria)
print(responseBodya, responseCodea, responseHeadersa)

print("Google UK Holidays (URL B) returns the following?")
local responseBodyb, responseCodeb, responseHeadersb = https.request(calendarUrib)
print(responseBodyb, responseCodeb, responseHeadersb)

If you run the code above, only the second Google (URL `B) returns anything..

[parkerc]

Arhhh there must be ‘redirects’ occuring…

And looking up http.request, it has a feature for that, which I’ve never used before, has anyone reading this ?

http.request{
  url = string,
  [sink = LTN12 sink,]
  [method = string,]
  [headers = header-table,]
  [source = LTN12 source],
  [step = LTN12 pump step,]
  [proxy = string,]
  [redirect = boolean,]
  [create = function]
}

[parkerc]

Checking the URL via Wheregoes.com, it suggests there are no redirects occuring, so that is perhaps not it..

Date Traced: 2021-11-21 11:23:32 GMT

User Agent: Wheregoes.com Redirect Checker/1.0

#CodeRequested URL
200https://ics.|fixtur.|es/v2/ipswich-town.|ics
Redirects: 0
1200https://ics.fixtur.es/v2/ipswich-town.ics
Trace Complete - 3 - text/calendar; charset=utf-8

[toggledbits]

If you're doing this in Vera firmware...

You may be dealing with the server demanding a level of encryption that the libraries cannot provide. This has been an increasingly common issue for many plugins, and Google was leading the charge in dropping weaker ciphers. In some of my own plugins, I've had to offer curl as an alternative mechanism, because the built-in curl uses different libraries and can be a workaround if the Lua libraries aren't cutting it.

It's also the case that the CA chain is now out of date on all Vera firmware, and as old CA certificates are now expiring, and Vera/EZlo has not yet produced firmware with upgraded CA stores, you could be running into a certificate validation failure due to the outdated or expired data. The -k on curl is a possible cure for this. In the Lua SSL library, you can set verify to none in your request table.

The issues have affected, in addition to my plugins (SiteSensor, Reactor for Vera, etc.) the Honeywell and MyQ plugins, and I think GCal3 has already had one round of retrofit as a result of Google's earlier changes. It's going to get worse, not better, unless they update the firmware (or we do...)

[parkerc]

@toggledbits said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

If you're doing this in Vera firmware...

I am indeed..

You may be dealing with the server demanding a level of
encryption that the libraries cannot provide. This has been an increasingly common issue for many plugins, and Google was leading the charge in dropping weaker ciphers. In some of my own plugins, I've had to offer curl as an alternative mechanism, because the built-in curl uses different libraries and can be a workaround if the Lua libraries aren't cutting it.

Thanks good to know., and ironically Google is the url that is working

It's also the case that the CA chain is now out of date on all Vera firmware, and as old CA certificates are now expiring, and Vera/EZlo has not yet produced firmware with upgraded CA stores, you could be running into a certificate validation failure due to the outdated or expired data. The -k on curl is a possible cure for this. In the Lua SSL library, you can set verify to none in your request table.

Ok, that verify and none option sounds an interesting approach. Does this look like a potential solution ?

lua https.request with certificate

I'm trying to make a request on lua with certificate. Recently I've gotten a COMODO SSL. I've tried many tutorials on the internet, but to no avail. I found this blog's proposal very interesting...

The issues have affected, in addition to my plugins (SiteSensor, Reactor for Vera, etc.) the Honeywell and MyQ plugins, and I think GCal3 has already had one round of retrofit as a result of Google's earlier changes. It's going to get worse, not better, unless they update the firmware (or we do...)

I’m really intrigued by your last point, how can we update the firmware ?

[toggledbits]

@parkerc said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

Does this look like a potential solution ?

More complicated than you need, probably. Setting verify to none works in most cases, at the expense of not detected a MITM attack (the likelihood of that is extremely low in my judgement, but don't substitute my judgement here for yours).

@parkerc said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

I’m really intrigued by your last point, how can we update the firmware ?

I have a build environment working for the version of the OS they are using, and I can build updated packages, sometimes. In a few cases, though, that entire world is just so old that building packages from current source into the old environment just doesn't work. But I have built an updated openssl and LuaSec and had some success with them. Really, Ezlo needs to step up and make it work, but I have a strong feeling they won't, and if it comes to that, I will circle back in that direction.

[parkerc]

@toggledbits said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

@parkerc said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

Does this look like a potential solution ?

More complicated than you need, probably. Setting verify to none works in most cases, at the expense of not detected a MITM attack (the likelihood of that is extremely low in my judgement, but don't substitute my judgement here for yours).

Sadly the verify = none option didn’t work, although it did at least returned a response code/status this time, when all previous attempts were blank..

require("socket")
local ssl = require("ssl")

-- TLS/SSL client parameters (omitted)
local params = {
  mode = "client",
  protocol = "tlsv1",
  verify = "none",
  options = "all",
}

local conn = socket.tcp()
conn:connect("https://ics.fixtur.es", 443)

-- TLS/SSL initialization
conn = ssl.wrap(conn, params)
conn:dohandshake()

conn:send("GET / HTTP/1.1\n\n")
local line, err = conn:receive()
print(err or line)
conn:close()

It returned : HTTP/1.1 400 Bad Request

I have a build environment working for the version of the OS they are using, and I can build updated packages, sometimes. In a few cases, though, that entire world is just so old that building packages from current source into the old environment just doesn't work. But I have built an updated openssl and LuaSec and had some success with them. Really, Ezlo needs to step up and make it work, but I have a strong feeling they won't, and if it comes to that, I will circle back in that direction.

I think you’re right, all their focus is on the new ezlo firmware, unless they refocus, or they open Vera up to the community to enhance, then I guess what we have from them is the best it gets

That said I’m now very intrigued by your ‘build’, I looked into openwrt and potential (re)build options and new packages, but that ended up being pretty damn complex route for me, especially considering all the things that have to be factored in, such as the kernel and processor (mips) etc.

Based one where I am now, I think my only option is to explore the more complex Lua coding route, and present a certificate myself with the following code.

But what cert / cipher details should I use ?

module("https", package.seeall) 

local socket = require "socket" 
local http = require "socket.http" 
local ssl = require "ssl" 
local ltn12 = require "ltn12" 

local try = socket.try 
local protect = socket.protect 

local DEFAULT_PROTOCOL = "sslv23" 
local DEFAULT_CAFILE = "/etc/ssl/certs/ca-certificates.crt" 
local DEFAULT_VERIFY = "peer" 
local DEFAULT_OPTIONS = "all" 
local DEFAULT_CIPHERS = "ADH-AES256-SHA:ADH-AES128-SHA:HIGH:MEDIUM" 
local DEFAULT_HTTPS_PORT = 443 

local https_mt = { 
    -- Create proxy functions for each call through the metatable 
    __index = function(tbl, key) 
        local f = function(prxy, ...) 
            local c = prxy.c 
            return c[key](c, ...) 
        end 
        tbl[key] = f    -- Save new proxy function in cache for speed 
        return f 
    end 
} 

local function new_create(params) 
    return function() 
        local t = { c = try(socket.tcp()) } 
        function t:connect(host, port) 
            try(self.c:connect(host, port)) 
            self.c = try(ssl.wrap(self.c, params)) 
            try(self.c:dohandshake()) 
            return 1 
        end 
        return setmetatable(t, https_mt) 
    end 
end 

local function request_generic(args) 
    local sslparams = { 
        mode = "client", 
        protocol = args.protocol or DEFAULT_PROTOCOL, 
        cafile = args.cafile or DEFAULT_CAFILE, 
        verify = args.verify or DEFAULT_VERIFY, 
        options = args.options or DEFAULT_OPTIONS, 
        ciphers = args.ciphers or DEFAULT_CIPHERS 
    } 
    local req = { 
      url = args.url, 
      port = args.port or DEFAULT_HTTPS_PORT, 
      sink = args.sink, 
      method = args.method, 
      headers = args.headers, 
      source = args.source, 
      step = args.step, 
      proxy = args.proxy,       -- Buggy? 
      redirect = args.redirect, 
      create = new_create(sslparams) 
    } 
    return http.request(req) 
end 

local function request_simple(url, body) 
    local tbl = { } 
    local req = { 
        url = url, 
        sink = ltn12.sink.table(tbl) 
    } 
    if body then 
        req.method = "POST" 
        req.source = ltn12.source.string(body) 
        req.headers = { 
            ["Content-length"] = #body, 
            ["Content-type"] = "application/x-www-form-urlencoded" 
        } 
    end 
    local _, status, headers = request_generic(req) 
    return table.concat(tbl), status, headers 
end 


function request(req_or_url, body) 
    if type(req_or_url) == "string" then 
        return request_simple(req_or_url, body) 
    else 
        return request_generic(req_or_url) 
    end 
end 

[toggledbits]

@parkerc said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

It returned : HTTP/1.1 400 Bad Request

If you are getting this far, the certificate and cipher issues are behind you, and it is now complaining about the content of the request. Since you are doing the request raw to a socket rather than using the http (or https) module, your request isn't complete; it's missing required headers. I'd start by using the http(s) module to make the request from this point.

@parkerc said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

Based one where I am now, I think my only option is to explore the more complex Lua coding route, and present a certificate myself with the following code.

I'm not sure that gains you anything. It may not improve the situation, either. The correct real fix here is to get the Vera's CA store up to date.

[parkerc]

Thanks @toggledbits , you actually beat me to it, I thought the 400 error was sign, so I tried this variation of http.request I had on file, and it worked !!

The ltn12.sink.table seemed to do the trick - by any chance do you know why ?

local http = require("ssl.https")
local ltn12 = require "ltn12"
r = {} -- init empty table
local res, code, headers, status = http.request{
url="https://ics.fixtur.es/v2/ipswich-town.ics",
sink = ltn12.sink.table( r )
}
print("status=".. tostring(status))
print("response= ".. table.concat( r, "" ) )

[toggledbits]

You're making a request to a different URL (different paths in the URL, specifically -- the previous request you showed had no path at all), which I'm sure had more (everything) to do with the difference in response than the ltn12.sink.table, which is nothing the client passes to the server, it's all just client-side handling for the response.

[parkerc]

@toggledbits said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

You're making a request to a different URL (different paths in the URL, specifically -- the previous request you showed had no path at all)

Sorry @toggledbits, , I’m not sure what you mean, putting the Google one to one side , both http.requests were made to the same url ""https://ics.fixtur.es/v2/ipswich-town.ics"" ?

[toggledbits]

@parkerc In this post the code that you posted shows:

local conn = socket.tcp()
conn:connect("https://ics.fixtur.es", 443)

-- TLS/SSL initialization
conn = ssl.wrap(conn, params)
conn:dohandshake()

conn:send("GET / HTTP/1.1\n\n")
local line, err = conn:receive()
print(err or line)
conn:close()

That's an HTTP get on the root (/) path (not /v2/ipswich.town.ics), and without full headers. Either is enough to cause a 400 from an API endpoint.

Your later post:

local res, code, headers, status = http.request{
url="https://ics.fixtur.es/v2/ipswich-town.ics",
sink = ltn12.sink.table( r )
}

That's using the full HTTP(S) module, so would be expected to have correct headers supplied by the library, and uses the longer path not used in the previous request/post. It was there you made the comment about ltn12.sink.table, which given that the request in the earlier post was both missing required headers and probably not for a valid path, and that the server is unaware of ltn12 or any handling on the client side of the response, seemed unrelated to the issue to me.

[parkerc]

@toggledbits said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

@parkerc In this post the code that you posted shows:

local conn = socket.tcp()
conn:connect("https://ics.fixtur.es", 443)

-- TLS/SSL initialization
conn = ssl.wrap(conn, params)
conn:dohandshake()

conn:send("GET / HTTP/1.1\n\n")
local line, err = conn:receive()
print(err or line)
conn:close()

That's an HTTP get on the root (/) path (not /v2/ipswich.town.ics), and without full headers. Either is enough to cause a 400 from an API endpoint.

My apologies ; you’re totally right, I forgot I tried the root address at one point too, it was one of the addresses I looked at to see if there were any redirects going on at that address..

Thanks again @toggledbits , all this has been yet another learning journey for me Admittedly it would be really nice if (one day) Vera could have an all encompassing http/web module, one that you could simply chuck any http and https request at (POST/GET), and it would know what to do and just work …

[toggledbits]

@parkerc said in Lua : http(s) request, works on some URLs, but not all ? But all work via the browser..:

Admittedly it would be really nice if (one day) Vera could have an all encompassing http/web module

Sadly, I don't think anything new will be coming to Vera from here forward...