Truncated web pages using wireguard over long distances
-
Set up:
a) Many many many many kms from home: laptop connected to modem router. Router running wireguard client to create a virtual network.
b) Home: modem router running wireguard server. openLuup pi4 connect to router and also a PC and other stuff, etc.
The problem: When accessing charts, AltUI or the openLupp console the web pages are returned OK up to the point where they are truncated and therefore fail to display anything useful.
Note this all works fine over short distances eg around a major city (I tested it) but not seemingly at world wide distances. ie network delays seem to be the issue here? Windows TeamViewer works fine overy the exact same network/wireguard set up. That's how I was able to get the openLuup logs shown below.
Here is any example of openLuup trying to return a chart:
2023-09-04 21:31:20.463 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x55aed35038 2023-09-04 21:31:20.464 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=316885191&Timeout=60&MinimumDelay=1500&_=1692128389970 HTTP/1.1 tcp{client}: 0x55ae538348 2023-09-04 21:31:20.465 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=316885191&Timeout=60&MinimumDelay=1500&_=1692129024374 HTTP/1.1 tcp{client}: 0x55addbe1e8 2023-09-04 21:31:20.477 openLuup.server:: GET /data_request?id=lr_render&target={temp_first_floor.w,temp_ground_floor.w,temp_back_wall_of_office.w,temp_inside_roof.w,temp_jps_bedrm_north.w,temp_outside.w}&title=Temperatures&height=750&from=-y&yMin=0&yMax=40 HTTP/1.1 tcp{client}: 0x55aed35038 2023-09-04 21:31:20.478 luup_log:6: DataGraph: drawing mode: connected, draw nulls as: null 2023-09-04 21:31:20.502 luup_log:6: DataGraph: Whisper query: CPU = 23.122 mS for 2016 points 2023-09-04 21:31:20.532 luup_log:6: DataGraph: Whisper query: CPU = 22.952 mS for 2016 points 2023-09-04 21:31:20.561 luup_log:6: DataGraph: Whisper query: CPU = 22.738 mS for 2016 points 2023-09-04 21:31:20.575 luup_log:6: DataGraph: Whisper query: CPU = 9.547 mS for 2016 points 2023-09-04 21:31:20.587 luup_log:6: DataGraph: Whisper query: CPU = 9.569 mS for 2016 points 2023-09-04 21:31:20.598 luup_log:6: DataGraph: Whisper query: CPU = 9.299 mS for 2016 points 2023-09-04 21:31:20.654 luup_log:6: visualization: LineChart(2016x7) 196kB in 51mS 2023-09-04 21:31:20.655 luup_log:6: DataGraph: render: CPU = 51.219 mS for 6x2016=12096 points 2023-09-04 21:31:20.755 openLuup.server:: error 'socket.select() not ready to send tcp{client}: 0x55aed35038' sending 2 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:20.855 openLuup.server:: error 'socket.select() not ready to send tcp{client}: 0x55aed35038' sending 6 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.037 openLuup.server:: error 'socket.select() not ready to send tcp{client}: 0x55aed35038' sending 2 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.138 openLuup.server:: error 'socket.select() not ready to send tcp{client}: 0x55aed35038' sending 6 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.332 openLuup.server:: error 'socket.select() not ready to send tcp{client}: 0x55aed35038' sending 2 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.432 openLuup.server:: error 'socket.select() not ready to send tcp{client}: 0x55aed35038' sending 6 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.507 openLuup.server:: error 'closed' sending 196367 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.507 openLuup.server:: ...only 144000 bytes sent 2023-09-04 21:31:21.507 openLuup.server:: error 'closed' sending 2 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.507 openLuup.server:: ...only 0 bytes sent 2023-09-04 21:31:21.507 openLuup.server:: error 'closed' sending 5 bytes to tcp{client}: 0x55aed35038 2023-09-04 21:31:21.507 openLuup.server:: ...only 0 bytes sent 2023-09-04 21:31:21.507 openLuup.server:: request completed (196367 bytes, 10 chunks, 1030 ms) tcp{client}: 0x55aed35038 2023-09-04 21:31:21.517 openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x55aed35038 2023-09-04 21:31:22.824 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x55aea22c88
Re: socket.select() not ready to send
Is there some sort of timeout I change; to see if this can make this work?
Note that openLuup is still running everything flawlessly for ages now, including the more recent addtions of ZigBee stuff. Much appreciated.
-
I’m no network expert, but I do note that openLuup uses chunked transfers for large amounts of data. I do, very occasionally, see the odd socket not ready error even on a LAN. This proved very hard to investigate, but I recall it being deeply embedded in issues within the Luasocket module. This doesn’t help you, but it begins to nudge my memory about the internal workings.
There are a couple of timeout parameters (something else which Luasocket is not very good at) which could be played with. I’ll take a look and get back on this.
-
The openLuup HTTP server has some user-configurable parameters:
{ Backlog = 2000, ChunkedLength = 16000, CloseIdleSocketAfter = 90, Port = 3480, SelectWait = 0.1 }
These can be changed with code in Lua Startup. The obvious one to try is SelectWait, but ChunkedLength might be relevant too. I'd suggest trying waits up to, say, a few seconds, and halving or doubling the chunk length.
Curiously, the syntax I'd recommend is
luup.attr_get "openLuup.HTTP" .SelectWait = 0.5
But, equally, something like:
luup.attr_set("openLuup.HTTP.SelectWait", 0.5)
should work too (the difference is whether the parameter is stored as a string or a number.)
There are other things to play with, but more deeply embedded. However, this indicates something a bit fragile somewhere along the line.
-
I looked at the ping times I was getting to remote servers using the likes of speedtest.net etc
On one speed tester I was getting pings arounf 250 msec, on another about 490 msec.
To send a message around half the world and back takes:
40075/300000=136 msecSo put in a few indirect cable runs and misc switches & buffers (but hopefully no satellites) then you can imagine ping times can get rather high. Window's TeamViewer is definately a bit laggy.
So inserted this into the start up code and then restarted openLuup:
luup.attr_set("openLuup.HTTP.SelectWait", 0.5)
and it all started working - so for the log example above I now get:
2023-09-06 21:30:13.155 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559b738498 2023-09-06 21:30:13.156 openLuup.server:: GET /data_request?id=lr_render&target={temp_first_floor.w,temp_ground_floor.w,temp_back_wall_of_office.w,temp_inside_roof.w,temp_jps_bedrm_north.w,temp_outside.w}&title=Temperatures&height=750&from=-y&yMin=0&yMax=40 HTTP/1.1 tcp{client}: 0x559b738498 2023-09-06 21:30:13.158 luup_log:6: DataGraph: drawing mode: connected, draw nulls as: null 2023-09-06 21:30:13.168 luup_log:6: DataGraph: Whisper query: CPU = 10.703 mS for 2016 points 2023-09-06 21:30:13.184 luup_log:6: DataGraph: Whisper query: CPU = 10.810 mS for 2016 points 2023-09-06 21:30:13.197 luup_log:6: DataGraph: Whisper query: CPU = 10.616 mS for 2016 points 2023-09-06 21:30:13.209 luup_log:6: DataGraph: Whisper query: CPU = 10.168 mS for 2016 points 2023-09-06 21:30:13.222 luup_log:6: DataGraph: Whisper query: CPU = 9.872 mS for 2016 points 2023-09-06 21:30:13.234 luup_log:6: DataGraph: Whisper query: CPU = 9.746 mS for 2016 points 2023-09-06 21:30:13.299 luup_log:6: visualization: LineChart(2016x7) 196kB in 60mS 2023-09-06 21:30:13.299 luup_log:6: DataGraph: render: CPU = 59.805 mS for 6x2016=12096 points 2023-09-06 21:30:14.086 openLuup.server:: request completed (196300 bytes, 13 chunks, 929 ms) tcp{client}: 0x559b738498
Not sure what the optimum value for the delay should be - I'm not keen to mess around with it remotely. Does the above log suggest a value? Or this log below:
Here's AltUI being loaded:
2023-09-06 21:49:23.723 luup.variable_set:: 10071.urn:micasaverde-com:serviceId:ZWaveNetwork1.LastPollSuccess was: 1694000093 now: 1694000963 #hooks:0 2023-09-06 21:49:23.836 openLuup.server:: request completed (6975 bytes, 1 chunks, 1700 ms) tcp{client}: 0x559aa2ee28 2023-09-06 21:49:23.957 openLuup.server:: request completed (6975 bytes, 1 chunks, 1146 ms) tcp{client}: 0x559bb50cb8 2023-09-06 21:49:24.217 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748913&Timeout=60&MinimumDelay=1500&_=1694000029638 HTTP/1.1 tcp{client}: 0x559aa2ee28 2023-09-06 21:49:24.979 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748913&Timeout=60&MinimumDelay=1500&_=1692128444563 HTTP/1.1 tcp{client}: 0x559bb50cb8 2023-09-06 21:49:25.158 openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x559aa2ee28 2023-09-06 21:49:25.428 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559b19a458 2023-09-06 21:49:25.430 openLuup.server:: GET /data_request?id=lr_ALTUI_Handler&command=home HTTP/1.1 tcp{client}: 0x559b19a458 2023-09-06 21:49:25.449 openLuup.server:: request completed (19761 bytes, 2 chunks, 19 ms) tcp{client}: 0x559b19a458 2023-09-06 21:49:25.951 openLuup.server:: GET /J_ALTUI_jquery.ui.touch-punch.min.js HTTP/1.1 tcp{client}: 0x559b19a458 2023-09-06 21:49:26.210 openLuup.server:: request completed (31597 bytes, 2 chunks, 258 ms) tcp{client}: 0x559b19a458 2023-09-06 21:49:26.221 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559c240428 2023-09-06 21:49:26.222 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559a9b66a8 2023-09-06 21:49:26.222 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559a49aec8 2023-09-06 21:49:26.222 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559bb54098 2023-09-06 21:49:26.223 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:26.224 openLuup.server:: GET /J_ALTUI_utils.js HTTP/1.1 tcp{client}: 0x559c240428 2023-09-06 21:49:26.970 openLuup.server:: request completed (202128 bytes, 13 chunks, 745 ms) tcp{client}: 0x559c240428 2023-09-06 21:49:26.970 openLuup.server:: GET /J_ALTUI_api.js HTTP/1.1 tcp{client}: 0x559a9b66a8 2023-09-06 21:49:27.498 openLuup.server:: request completed (55724 bytes, 4 chunks, 527 ms) tcp{client}: 0x559a9b66a8 2023-09-06 21:49:27.499 openLuup.server:: GET /J_ALTUI_upnp.js HTTP/1.1 tcp{client}: 0x559a49aec8 2023-09-06 21:49:27.760 openLuup.server:: request completed (22906 bytes, 2 chunks, 261 ms) tcp{client}: 0x559a49aec8 2023-09-06 21:49:27.761 openLuup.server:: GET /J_ALTUI_verabox.js HTTP/1.1 tcp{client}: 0x559bb54098 2023-09-06 21:49:28.500 openLuup.server:: request completed (97222 bytes, 7 chunks, 739 ms) tcp{client}: 0x559bb54098 2023-09-06 21:49:28.501 openLuup.server:: GET /J_ALTUI_multibox.js HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:28.753 openLuup.server:: request completed (40237 bytes, 3 chunks, 251 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:28.764 openLuup.server:: GET /J_ALTUI_plugins.js HTTP/1.1 tcp{client}: 0x559b19a458 2023-09-06 21:49:29.532 openLuup.server:: request completed (231053 bytes, 15 chunks, 767 ms) tcp{client}: 0x559b19a458 2023-09-06 21:49:29.532 openLuup.server:: GET /J_ALTUI_iphone.js HTTP/1.1 tcp{client}: 0x559c240428 2023-09-06 21:49:29.535 openLuup.server:: request completed (22770 bytes, 2 chunks, 1 ms) tcp{client}: 0x559c240428 2023-09-06 21:49:29.536 openLuup.server:: GET /J_ALTUI_uimgr.js HTTP/1.1 tcp{client}: 0x559a9b66a8 2023-09-06 21:49:30.827 openLuup.server:: request completed (717146 bytes, 45 chunks, 1291 ms) tcp{client}: 0x559a9b66a8 2023-09-06 21:49:30.840 openLuup.server:: GET /data_request?id=variableget&DeviceNum=0&serviceId=urn:micasaverde-com:serviceId:HomeAutomationGateway1&Variable=Mode&_=1692128444564 HTTP/1.1 tcp{client}: 0x559b6dee78 2023-09-06 21:49:30.943 openLuup.server:: request completed (1 bytes, 1 chunks, 102 ms) tcp{client}: 0x559b6dee78 2023-09-06 21:49:31.281 openLuup.server:: GET /favicon.ico HTTP/1.1 tcp{client}: 0x559b19a458 2023-09-06 21:49:31.283 openLuup.server:: request completed (15086 bytes, 0 chunks, 1 ms) tcp{client}: 0x559b19a458 2023-09-06 21:49:31.284 openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x559b19a458 2023-09-06 21:49:31.451 openLuup.server:: GET /data_request?id=user_data&output_format=json&DataVersion=undefined&_=1694000965692 HTTP/1.1 tcp{client}: 0x559a9b66a8 2023-09-06 21:49:32.569 openLuup.server:: request completed (1560409 bytes, 98 chunks, 1117 ms) tcp{client}: 0x559a9b66a8 2023-09-06 21:49:34.392 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x5599e944b8 2023-09-06 21:49:34.392 openLuup.io.server:: HTTP:3480 connection from 10.0.0.2 tcp{client}: 0x559a7b2ba8 2023-09-06 21:49:34.392 openLuup.server:: GET /luvd/D_GenericZigbeeDevice.xml HTTP/1.1 tcp{client}: 0x559c240428 2023-09-06 21:49:34.393 openLuup.server:: request completed (299 bytes, 0 chunks, 0 ms) tcp{client}: 0x559c240428 2023-09-06 21:49:34.393 openLuup.server:: GET /luvd/D_openLuup.xml HTTP/1.1 tcp{client}: 0x559a9b66a8 2023-09-06 21:49:34.393 openLuup.server:: request completed (754 bytes, 0 chunks, 0 ms) tcp{client}: 0x559a9b66a8 2023-09-06 21:49:34.393 openLuup.server:: GET /luvd/D_InfoViewer1.xml HTTP/1.1 tcp{client}: 0x559a49aec8 2023-09-06 21:49:34.393 openLuup.servlet:: file not found:D_InfoViewer1.xml 2023-09-06 21:49:34.394 openLuup.server:: request completed (132 bytes, 0 chunks, 0 ms) tcp{client}: 0x559a49aec8 2023-09-06 21:49:34.394 openLuup.server:: GET /luvd/D_BinaryLight1.xml HTTP/1.1 tcp{client}: 0x559bb54098 2023-09-06 21:49:34.394 openLuup.server:: request completed (1031 bytes, 0 chunks, 0 ms) tcp{client}: 0x559bb54098 2023-09-06 21:49:34.394 openLuup.server:: GET /luvd/D_SmartHomeHelper.xml HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:34.394 openLuup.servlet:: file not found:D_SmartHomeHelper.xml 2023-09-06 21:49:34.395 openLuup.server:: request completed (136 bytes, 0 chunks, 0 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:34.404 openLuup.server:: GET /data_request?id=lr_ALTUI_Handler&command=getCustomPages&_=1694000965693 HTTP/1.1 tcp{client}: 0x5599e944b8 2023-09-06 21:49:34.405 openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x559c240428 2023-09-06 21:49:34.405 openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x559a9b66a8 2023-09-06 21:49:34.405 openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x559bb54098 2023-09-06 21:49:34.405 openLuup.server:: GET /data_request?id=lr_ALTUI_Handler&command=getWorkflows&_=1694000965694 HTTP/1.1 tcp{client}: 0x559a7b2ba8 2023-09-06 21:49:34.507 openLuup.server:: request completed (2 bytes, 1 chunks, 102 ms) tcp{client}: 0x5599e944b8 2023-09-06 21:49:34.507 openLuup.server:: request completed (2 bytes, 1 chunks, 101 ms) tcp{client}: 0x559a7b2ba8 2023-09-06 21:49:34.807 openLuup.server:: GET /data_request?id=variableget&DeviceNum=0&serviceId=urn:micasaverde-com:serviceId:HomeAutomationGateway1&Variable=Mode&_=1694000965697 HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:34.908 openLuup.server:: request completed (1 bytes, 1 chunks, 101 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:35.613 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=1&Timeout=60&MinimumDelay=1500&_=1694000965698 HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:37.431 openLuup.server:: request completed (882908 bytes, 56 chunks, 1817 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:37.442 luup.variable_set:: 10191.urn:micasaverde-com:serviceId:EnergyMetering1.KWHReading was: 1694000916 now: 1694000976 #hooks:0 2023-09-06 21:49:37.454 openLuup.server:: request completed (6972 bytes, 1 chunks, 12474 ms) tcp{client}: 0x559bb50cb8 2023-09-06 21:49:37.982 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748913&Timeout=60&MinimumDelay=1500&_=1694000965699 HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:38.095 openLuup.server:: request completed (6972 bytes, 1 chunks, 112 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:38.468 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748915&Timeout=60&MinimumDelay=1500&_=1694000965700 HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:38.486 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748915&Timeout=60&MinimumDelay=1500&_=1692128444565 HTTP/1.1 tcp{client}: 0x559bb50cb8 2023-09-06 21:49:39.406 luup.zigbee2mqtt:0: Topic ignored: zigbee2mqtt/bridge/logging 2023-09-06 21:49:39.733 openLuup.server:: request completed (7069 bytes, 1 chunks, 1264 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:39.744 openLuup.server:: request completed (7069 bytes, 1 chunks, 1257 ms) tcp{client}: 0x559bb50cb8 2023-09-06 21:49:40.130 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748918&Timeout=60&MinimumDelay=1500&_=1694000965701 HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:40.771 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748918&Timeout=60&MinimumDelay=1500&_=1692128444566 HTTP/1.1 tcp{client}: 0x559bb50cb8 2023-09-06 21:49:41.819 openLuup.server:: GET /data_request?id=variableget&DeviceNum=0&serviceId=urn:micasaverde-com:serviceId:HomeAutomationGateway1&Variable=Mode&_=1692128444567 HTTP/1.1 tcp{client}: 0x559b6dee78 2023-09-06 21:49:41.922 openLuup.server:: request completed (1 bytes, 1 chunks, 102 ms) tcp{client}: 0x559b6dee78 2023-09-06 21:49:45.188 openLuup.server:: GET /data_request?id=variableget&DeviceNum=0&serviceId=urn:micasaverde-com:serviceId:HomeAutomationGateway1&Variable=Mode&_=1694000965702 HTTP/1.1 tcp{client}: 0x559a49aec8 2023-09-06 21:49:45.292 openLuup.server:: request completed (1 bytes, 1 chunks, 102 ms) tcp{client}: 0x559a49aec8 2023-09-06 21:49:48.690 luup.variable_set:: 10224.urn:micasaverde-com:serviceId:EnergyMetering1.KWHReading was: 1694000688 now: 1694000988 #hooks:0 2023-09-06 21:49:48.702 openLuup.server:: request completed (6972 bytes, 1 chunks, 7930 ms) tcp{client}: 0x559bb50cb8 2023-09-06 21:49:49.127 openLuup.server:: request completed (6972 bytes, 1 chunks, 8996 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:49.725 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748920&Timeout=60&MinimumDelay=1500&_=1692128444568 HTTP/1.1 tcp{client}: 0x559bb50cb8 2023-09-06 21:49:50.410 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748920&Timeout=60&MinimumDelay=1500&_=1694000965703 HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:51.623 luup.variable_set:: 10149.urn:micasaverde-com:serviceId:EnergyMetering1.KWHReading was: 1694000944 now: 1694000991 #hooks:0 2023-09-06 21:49:52.049 openLuup.server:: request completed (6972 bytes, 1 chunks, 2323 ms) tcp{client}: 0x559bb50cb8 2023-09-06 21:49:52.060 openLuup.server:: request completed (6972 bytes, 1 chunks, 1649 ms) tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:52.428 openLuup.server:: GET /data_request?id=lu_status2&output_format=json&DataVersion=999748922&Timeout=60&MinimumDelay=1500&_=1694000965704 HTTP/1.1 tcp{client}: 0x559bf9b2d8 2023-09-06 21:49:52.460 openLuup.server:: GET /console?page=log HTTP/1.1 tcp{client}: 0x559bb50cb8
Anyway good result. I can now control anything that needs controlling. Ak -thanks for the help once again.