ssl stuff: curl instead of copy sessiondir to worker node scratch dir
As I have mentioned on skype I am working on setting up ARC cache in the OpenStack grid cluster (UIO_CLOUD).
Since I do not want to nfs mount the ARC caches onto the clusters worker nodes I will have to fetch the input files in the sessiondirectory via curl. So the idea is to modify the submission script to use curl instead of just copy.
I have gotten dummy self signed chain to work on a test nginx server. Basically something like this
curl -v --insecure --cert ./client.pem --key ./client.pem https://158.39.48.46/sessiondir/
Likewise, if I do cat client.pem client.key > bundle.pem
this also works:
curl -v --insecure --cert ./bundle.pem https://158.39.48.46/sessiondir/
i.e. I will be denied access if I have not issued the client certificate, and will be given access if I do (as in the example above). For this simple first test I have simply created my own ca-cert, and from that the server and client certificates. That was the easy dummy setup just getting the syntax right for ssl verification of the client on the server.
For ARC with many CA's I have to cocatenate all CA-pem files into one since nginx only takes a file not a directory containing the certificates.
I have tested with two ca's on the test-server cocatenated into one and it all works well.
Now the question is how to get this right in the ARC case.
- I have cocatenated all CA pem files into one cat /etc/grid-security/*pem > allcas.pem
- I have created a proxy certificate with arcproxy
- I try curl -v --cert /tmp/x509up_u1000 https://frontend001.grid.uiocloud.no:8443 --insecure
Might be that there are some intermediate certificates I am missing, I see the below in the /var/log/nginx/error.log
**So main question is: any clue what is the problem and how to solve? **
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL handshake handler: 0
2019/10/09 22:51:48 [debug] 25846#0: *2 verify:0, error:20, depth:0, subject:"/DC=org/DC=terena/DC=tcs/C=NO/O=Universitetet i Oslo/CN=Maiken Pedersen/CN=173828700", issuer:"/DC=org/DC=terena/DC=tcs/C=NO/O=Universitetet i Oslo/CN=Maiken Pedersen"
2019/10/09 22:51:48 [debug] 25846#0: *2 verify:0, error:40, depth:0, subject:"/DC=org/DC=terena/DC=tcs/C=NO/O=Universitetet i Oslo/CN=Maiken Pedersen/CN=173828700", issuer:"/DC=org/DC=terena/DC=tcs/C=NO/O=Universitetet i Oslo/CN=Maiken Pedersen"
2019/10/09 22:51:48 [debug] 25846#0: *2 verify:0, error:21, depth:0, subject:"/DC=org/DC=terena/DC=tcs/C=NO/O=Universitetet i Oslo/CN=Maiken Pedersen/CN=173828700", issuer:"/DC=org/DC=terena/DC=tcs/C=NO/O=Universitetet i Oslo/CN=Maiken Pedersen"
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL_do_handshake: 1
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2019/10/09 22:51:48 [debug] 25846#0: *2 reusable connection: 1
2019/10/09 22:51:48 [debug] 25846#0: *2 http wait request handler
2019/10/09 22:51:48 [debug] 25846#0: *2 malloc: 000055EACE82E860:1024
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL_read: -1
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL_get_error: 2
2019/10/09 22:51:48 [debug] 25846#0: *2 free: 000055EACE82E860
2019/10/09 22:51:48 [debug] 25846#0: timer delta: 7
2019/10/09 22:51:48 [debug] 25846#0: worker cycle
2019/10/09 22:51:48 [debug] 25846#0: epoll timer: 59881
2019/10/09 22:51:48 [debug] 25846#0: epoll: fd:3 ev:0001 d:00007F8900E8A1E1
2019/10/09 22:51:48 [debug] 25846#0: *2 http wait request handler
2019/10/09 22:51:48 [debug] 25846#0: *2 malloc: 000055EACE82E860:1024
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL_read: 122
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL_read: -1
2019/10/09 22:51:48 [debug] 25846#0: *2 SSL_get_error: 2
2019/10/09 22:51:48 [debug] 25846#0: *2 reusable connection: 0
2019/10/09 22:51:48 [debug] 25846#0: *2 posix_memalign: 000055EACE72C3C0:4096 @16
2019/10/09 22:51:48 [debug] 25846#0: *2 http process request line
2019/10/09 22:51:48 [debug] 25846#0: *2 http request line: "GET / HTTP/1.1"
2019/10/09 22:51:48 [debug] 25846#0: *2 http uri: "/"
2019/10/09 22:51:48 [debug] 25846#0: *2 http args: ""
2019/10/09 22:51:48 [debug] 25846#0: *2 http exten: ""
2019/10/09 22:51:48 [debug] 25846#0: *2 posix_memalign: 000055EACE8CEF70:4096 @16
2019/10/09 22:51:48 [debug] 25846#0: *2 http process request header line
2019/10/09 22:51:48 [debug] 25846#0: *2 http header: "Range: bytes=tial_chain"
2019/10/09 22:51:48 [debug] 25846#0: *2 http header: "User-Agent: curl/7.29.0"
2019/10/09 22:51:48 [debug] 25846#0: *2 http header: "Host: frontend001.grid.uiocloud.no:8443"
2019/10/09 22:51:48 [debug] 25846#0: *2 http header: "Accept: */*"
2019/10/09 22:51:48 [debug] 25846#0: *2 http header done
2019/10/09 22:51:48 [info] 25846#0: *2 client SSL certificate verify error: (21:unable to verify the first certificate) while reading client request headers, client: 158.39.48.56, server: frontend001.grid.uiocloud.no, request: "GET / HTTP/1.1", host: "frontend001.grid.uiocloud.no:8443"
2019/10/09 22:51:48 [debug] 25846#0: *2 http finalize request: 495, "/?" a:1, c:1
2019/10/09 22:51:48 [debug] 25846#0: *2 event timer del: 3: 1570654368495
2019/10/09 22:51:48 [debug] 25846#0: *2 http special response: 495, "/?"
2019/10/09 22:51:48 [debug] 25846#0: *2 http set discard body
2019/10/09 22:51:48 [debug] 25846#0: *2 xslt filter header
2019/10/09 22:51:48 [debug] 25846#0: *2 HTTP/1.1 400 Bad Request