Disaster recovery and virtualisation

[parkerc]

Hi

I’m looking to rework some code I found online so I can use it on Vera, and while I’ve managed to translate / convert a number of things, I’ve got stuck on a few things it does, around encryption/decryption etc.

All the original Lua code is here - > (https://forum.logicmachine.net/showthread.php?tid=232&pid=16580#pid16580 )

Plus there looks to be a python version too here - > (https://github.com/florianholzapfel/panasonic-viera/issues/9#issuecomment-476919658)

Here’s an extract of the code where AES 128 CBC is required?

function encrypt_soap_payload(data, key, hmac_key, iv)
payload = '000000000000'
  n = #data

payload = payload .. string.char(bit.band(bit.rshift(n, 24), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 16), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 8), 0xFF))
payload = payload .. string.char(bit.band(n, 0xFF))

payload = payload .. data

 aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 1) 
  ciphertext = aes_cbc:encrypt(payload)
  sig = encdec.hmacsha256(ciphertext, hmac_key, true)
  encrypted_payload = encdec.base64enc(ciphertext .. sig)
  return encrypted_payload
end

function decrypt_soap_payload(data, key, hmac_key, iv)
  aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 0) 
  decrypted = aes_cbc:decrypt(encdec.base64dec(data))
  decrypted = string.gsub(string.sub(lmcore.strtohex(decrypted), 33), '%x%x', function(value) return string.char(tonumber(value, 16)) end) 
  return decrypted
end

I can get to the point where I can create the parameters for the payload encrypt request (example below), it’s just the encryption/decryption I cant do..

data="1234"
key="\\S„ßÍ}/Ìa5!"
hmac_key="¹jz¹2¸F\r}òcžÎ„ 臧.ª˜¹=¤µæŸ"
iv=" {¬£áæ‚2žâ3ÐÞË€ú "

I’ve found this aes.lua (https://github.com/openresty/lua-resty-string/blob/master/lib/resty/aes.lua ) module online, but that requires loads of others modules too, most notably ffi.lua. Which I’d ideally like to avoid.

I also came across this aes128.lua (https://github.com/somesocks/lua-lockbox/blob/master/lockbox/cipher/aes128.lua ) but i’m not sure how that handles all the other parameters e.g specify the cbc aspect etc.

Finally there’s this aes256ecb.lua script (https://github.com/idiomic/Lua_AES/blob/master/AES.lua) , could that be converted to aes 128 cbc and then used in the above?

Any help/advice on this would be appreciated..

[parkerc]

I can see on Vera, that it has openssl installed, is that an option ?

root@MiOS_ 12345678:~# openssl version
OpenSSL 1.0.2l  25 May 2017

root@MiOS_ 12345678:~# openssl ciphers
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:DH-DSS-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DH-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DH-RSA-AES256-SHA256:DH-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DH-RSA-AES256-SHA:DH-DSS-AES256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:DH-DSS-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:DH-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DH-RSA-AES128-SHA256:DH-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DH-RSA-AES128-SHA:DH-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DH-RSA-SEED-SHA:DH-DSS-SEED-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SRP-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:SRP-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DH-RSA-DES-CBC3-SHA:DH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA

There seems a number of related posts online - where similar attempts have been made e.g. https://stackoverflow.com/questions/65918428/how-to-provide-string-iv-and-key-to-openssl-decrypt-command

Never having done anything like this before, looking for some support/guidance from anyone who understands all this cipher/encryption stuff much more than I

[toggledbits]

Absolutely OpenSSL is the way to go, rather than trying to implement the encryption in Lua. You can use os.execute() or io.popen() to run the command and capture the output. You're probably looking at some variant of:

    openssl aes-128-cbc -e -a -K hex-key -iv hex-iv [-in xxx]

[parkerc]

Thanks @toggledbits

I’ve been playing around with OpenSSL a little this morning, I’ve, not had much success so far., based on the source code, it looks like i need to encrypt the generated payload with the created key and iv using aes 128 cbc.. Does that sound right to you.. (Looking at OpenSSL guide, it seems pretty comprehensive and complex.- https://www.openssl.org/docs/man1.1.1/man1/openssl-enc.html )

Here’s my code so far, any thoughts/suggestions welcome..

local name = "openssl"
local version = "version"
local data="1234"
local key="\\S„ßÍ}/Ìa4!"
local hmac_key="¹jz¹2¸F\r}òcžÎ„ 臧.ª˜¹=¤µæŸ"
local iv=" {¬£áæ‚2žâ3ÐÞË€ú "
local payload = "0000000000001234"

--local buildsslcommand1 = name .." "..version
--local buildsslcommand2 = "openssl aes-128-cbc -e -a -K hex-"..key.." -iv hex-"..iv.."[-in xxx]"
--local buildsslcommand3 = "openssl enc -aes-128-cbc -iv "..iv.." -k "..key.." | hexdump -C"
local buildsslcommand = "openssl enc -aes-128-cbc -nosalt -e -a -A "..payload.." -K "..key.." -iv "..iv
print("Command to send = " ..buildsslcommand)

local file = assert(io.popen(buildsslcommand, 'r'))
local output = file:read('*all')
file:close()
print(string.len(output)) --> just count what's returned.
print(output) -- > Prints the output of the command.

[toggledbits]

Practice on the command line and don't worry about the Lua until you know how the command works and what you need to give it.

Your key and IV have to be hex, so that's your first challenge. The command will want to read stdin if you don't specify -in <file>; you can't put the payload on the command line directly. Plus it contains binary data, so that's a non-starter. So at a minimum you're going to be writing the payload to a temporary file and encrypting that with -in. OpenSSL can write the encrypted output to stdout (it will if you don't specify -out <file>), which is good because you can read that directly when using io.popen(), so that part's going to be OK.

[parkerc]

Thanks @toggledbits ,

To make the key and iv hex, could I use binascii.lua (https://github.com/tst2005/binascii/blob/master/binascii.lua) ?

—Converts a string of bytes to a hexadecimal string
local function hexlify(s)
	local a = {}
	for i=1,#s do
		local c = s:sub(i,i)
		local byte = c:byte()
		a[#a+1] = ('%02X'):format(byte)
	end
	return table.concat(a)
end

As for the OpenSSL command line structure, I’ve come across a number of different examples online - this one seems to relate to what you’re suggesting (I’d just need to change it to 128, rather than 256)

openssl enc -aes-256-cbc -nosalt -e -a -A -in input.dat -K '7c07f68ea8494b2f8b9fea297119350d78708afa69c1c76' -iv 'FEDCBA987654321' -out input-test.enc

QQ : You mentioned that I’m going to need to write the payload to a temp file and encrypt it with -in?

How does -in do any encryption?

In the OpenSSL manual it just says the following about -in and -out

-in filename
The input filename, standard input by default.

-out filename
The output filename, standard output by default.

[toggledbits]

@parkerc said in Lua - Code to encrypt / decrypt with AES 128 CBC:

QQ : You mentioned that I’m going to need to write the payload to a temp file and encrypt it with -in?

You need to use -in on the encryption command to tell it to read from a file. You didn't have -in in your previous example, you just put the payload on the command line, and that doesn't work.

The hex conversion doesn't need two steps for the byte conversion and math for target array position:

-- Converts a string of bytes to a hexadecimal string
local function hexlify(s)
	local a = {}
	for i=1,#s do
		a[i] = ('%02X'):format( s:byte( i ) )
	end
	return table.concat(a)
end

[parkerc]

Ok, not sure how far I’ve progressed with this, but hopefully I’m getting somewhere ??

Here are the values created earlier..

local key= "\\S„ßÍ}/Ìa4!"
local iv = "{¬£áæ‚2žâ3ÐÞË€ú"
local payload = "0000000000001234"

I’ve put the payload value into a file..

local file = "etc/payload.txt"
local outf = io.open(file, "w")
outf:write(payload)		
outf:close()

Then created Hex values of the key and iv

local function hexlify(s)
	local a = {}
	for i=1,#s do
		a[i] = ('%02X'):format( s:byte( i ) )
	end
	return table.concat(a)
end

print(hexlify(key))
print(hexlify(iv))

keyHEX = "5C53E2809EC39FC38D7D191E2FC38C1D61057F3421"
ivHEX = "7BC2ACC2A3C3A1C3A6E2809A32C5BEC3A233C390C39EC38BE282ACC3BA"

Then I’ve taken those new hex values and accessed the command line of vera to run the following openssl command I created…

openssl enc -aes-128-cbc -nosalt -e -a -A -in etc/payload.txt -K '5C53E2809EC39FC38D7D191E2FC38C1D61057F3421' -iv '7BC2ACC2A3C3A1C3A6E2809A32C5BEC3A233C390C39EC38BE282ACC3BA' -out etc/payload2.txt

The above resulted in etc/payload2.txt being created, however it's empty and the command line returns the following messages.

hex string is too long
invalid hex iv value

Other than checking how the initial key and I’ve values are created again, is there anything else I need to look into/check ?

[toggledbits]

Those initial values for key and IV being binary data in strings could be a problem. Your editor may be wrecking them because it thinks they are Unicode, or they've already been wrecked along the way by other means. Both are supposed to be 16 bytes, and you've got 29 for IV and 21 for the key, so something has definitely gone wrong, and Unicode/UTF-8 encoding and decoding in the file handling is a prime suspect.

Fortunately, the IV doesn't really matter, I think. You could just use the first 16 bytes of the payload for testing. But it's supposed to be a 16-byte (128 bit) nonce, used only one time with the key (so if you encrypt something else with the same key, you should use a different IV). Maybe just generate 16 random bytes and call it good, but make sure to seed the random number generator, otherwise it generates the same 16 "random" numbers after every restart (that's a feature, actually, but you have to remember to seed in production to get away from it).

And really, probably the key doesn't matter either. You need both the key and IV to encrypt and decrypt, so it's not like you are dealing with a known key from another system like a remote API (unless they are doing this all wrong and using the same key and IV for every payload--that would be an... error). Pick 16 bytes and go to town!

patrick@drupal:/tmp$ echo "I am a secret message." > in.txt
patrick@drupal:/tmp$ cat in.txt
I am a secret message.

# Encrypt
patrick@drupal:/tmp$ openssl aes-128-cbc -e -a -in in.txt -K '9988227744aaff003388ffccee1188ff' -iv '112233445566778899aabbccddeeff00' -out out.txt
patrick@drupal:/tmp$ cat out.txt
MQi9QVm1/R3dvZncbX0nQeRqlf0+2oFcMuW/vp0FQ2Q=

# Now decrypt
patrick@drupal:/tmp$ openssl aes-128-cbc -d -a -in out.txt -K '9988227744aaff003388ffccee1188ff' -iv '112233445566778899aabbccddeeff00'
I am a secret message.

[parkerc]

Thanks @toggledbits

I think you’re right about the Lua/Luup editor wrecking the format of the values being generated, plus I’m perhaps not helping matters by working on various parts of the overall code separately ; so I have been copying things over.

FYI - You can see how the generated values are stored/presented differently (one image is via a print screen using LuaView and the other is writing them as new variables into Vera)

I’m going to try and do as much as I can using the Vera written values, as ultimately that’s where my target for where this code will need to eventually run..

[toggledbits]

OK, yeah, so big problem, you cannot store binary data in state variables. That is going to fail, 100%. State variables are character strings, and assumed to contain character data, and when they are stored, they are put through UTF-8 encoding and decoding. Your binary data is going to look like Unicode characters to the innards of Luup throwing this stuff around, and anything that isn't a valid codepoint is going to get changed to something else (and boom, data corrupted), while converting it to JSON to store on flash and back, etc.

Store everything in hex or base64.

[parkerc]

What would I do without you

Ok, so I can still write to a device variable in Vera, I’d just need to encode it first , in either one of those two options. e.g like this if base64?

local function base64_encode(data)
	local b='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
    return ((data:gsub('.', function(x) 
        local r,b='',x:byte()
        for i=8,1,-1 do r=r..(b%2^i-b%2^(i-1)>0 and '1' or '0') end
        return r;
    end)..'0000'):gsub('%d%d%d?%d?%d?%d?', function(x)
        if (#x < 6) then return '' end
        local c=0
        for i=1,6 do c=c+(x:sub(i,i)=='1' and 2^(6-i) or 0) end
        return b:sub(c+1,c+1)
    end)..({ '', '==', '=' })[#data%3+1])
end

local key = string.char(unpack(key_vals)) - - 
local keyEnc = base64_encode(key)
luup.variable_set("urn:nodecentral-net:serviceId:NodeCentral1","key",keyEnc, 1187)

[toggledbits]

It's easier than that in Luup...

local mime = require("mime")
local encoded = mime.b64( "Hello world!" );
local decoded = mime.unb64( encoded );
luup.log( decoded, 2 )

[parkerc]

Awesome, thanks - I often forget what’s already there in luup.. !

Also am I right in saying that i only need to enc/dec in hex or base64 if I’m planning to store (read/write) those values/variables, if I can keep them ‘active; and flowing within the code itself (in memory) then, I’m ok ?

[toggledbits]

Yes, in memory is fine. So for example, you may generate IV from random numbers byte by byte, put it through b64 or hex before storing it on a state variable. For openssl, you'll be giving key and IV in hex, so that seems a sensible way to keep it. Once it's converted to hex, you'll never need to reverse that, since it can be stored in a state variable as hex, and can be handed directly to openssl as hex.

[parkerc]

This time round, I seem to have generated some new hex values this time, which I’ve used in a io.popen openssl command, feels a bit better, but sadly not quite yet right (yet).. FYI

openssl enc -aes-128-cbc -nosalt -e -a -A -in etc/payload.txt -K 'enNIQi9wNmYxd1RDUXMvNHJhdHNGZz09' -iv 'B6B1919A88C696AE87A692B1BBADCE9B' -out etc/payload2.txt
non-hex digit
invalid hex key value

[toggledbits]

Well the key you show is definitely not hex, because 16 bytes of hex should be 32 characters long with only 0-9, A-F. Looks like you have base64 for the key, not hex.

[parkerc]

Progress at last ! well I’ve at least got the key and iv in the correct hex format for the OpenSSL aes cbc encryption command to work.🥳

Which leads me on to the next part, where the output of the above is referred to as the ciphertext;, of which I now need to do the following with..

  ciphertext = MyopensslAEScbcCall()
  sig = encdec.hmacsha256(ciphertext, hmac_key, true)
  encrypted_payload = encdec.base64enc(ciphertext .. sig)
  return encrypted_payload

hmacsha256 seems to be next, and it might also be something I could do with OpenSSL too ? -> https://unix.stackexchange.com/questions/610039/how-to-do-hmacsha256-using-openssl-from-terminal ?

I also found this too - https://github.com/jqqqi/Lua-HMAC-SHA256/blob/master/sha256.lua

@toggledbits any suggestions/recommendations on what to do with this next part ?

[toggledbits]

I would stick with OpenSSL. You just need to assemble the correct bits (HMAC key and ciphertext) to send to the hash function, compute the hash, then append the hash to the ciphertext. The trick here will be that you currently have the AES encyption outputting its result base64-encoded... you can't append the signature to that. You need the ciphertext output in binary for that, so you need to remove the -a (and -A) to let the output be binary. Then you can put that into the hash. And then you can take the (binary) hash output and append that to the binary ciphertext, and then you base64 encode the assembled result at the very end. The whole thing takes a few steps. I would recommend writing a shell script to do all the steps, and just have Lua run the shell script. You can then more easily develop and test the steps (because they are all in the shell script and runnable from the command line).

[parkerc]

Thanks @toggledbits - I struggle enough with Lua, so i admire your optimism in thinking I could create a shell script too

I must admit, I can’t help but feel like I’m making this more complicated that it needs to be, especially as there are working Lua and Python scripts already out there, which seem to be doing everything natively (although I do admit much of this thread is trying to plug holes in Vera )

With that said, the following might help you understand the bigger picture of what I’m trying to replicate, and maybe help me more, as I found some related posts, the first one which uses python e.g. https://github.com/florianholzapfel/panasonic-viera/issues/9 - and I’ve extracted the related HMAC part below.

…
# Let's encrypt it with AES-CBC! We need to make sure we pad it to a multiple of 16 bytes beforehand
aes = AES.new(key, AES.MODE_CBC, iv)
ciphertext = aes.encrypt(pad(payload))

# Calculate the HMAC-SHA-256 signature of our encrypted payload
sig = hmac.new(hmac_key, ciphertext, hashlib.sha256).digest()

# Concatenate the HMAC signature to the encrypted payload and base64 encode it, and we're done!
encrypted_payload = base64.b64encode(ciphertext + sig)

Plus there’s this pure Lua version called Haslib -> https://github.com/howmanysmall/Rewrites/blob/master/src/HashLib.lua - which looks like it might be promising to add to Vera as a module ?

USAGE:
	Input data should be a string
	Result (SHA digest) is returned in hexadecimal representation as a string of lowercase hex digits.

	Simplest usage example:
		local HashLib = require(script.HashLib)
		local your_hash = HashLib.sha256("your string")

Looking at the functions within the latter HashLib.lua file, it has one called local function hmac(hash_func, key, message) which makes me wonder could i use that and do the following.. ?

local HashLib = require(script.HashLib)
local mySig = HashLib.hmac(sha256, ciphertext, hmac_key)