java - SSL Handshaking Using Self-Signed Certs and SSLEngine (JSSE) -

i have been tasked implement custom/standalone java webserver can process ssl , non-ssl messages on same port.

i have implemented nio server , working quite non-ssl requests. having heck of time ssl piece , use guidance.

here's have done far.

in order distinguish between ssl , non-ssl messages, check first byte of inbound request see if ssl/tls message. example:

   byte = read(buf);    if (totalbytesread==1 && (a>19 && a<25)){        parsetls(buf);    } 

in parsetls() method instantiate sslengine this:

   java.security.keystore ks = java.security.keystore.getinstance("jks");    java.security.keystore ts = java.security.keystore.getinstance("jks");     ks.load(new java.io.fileinputstream(keystorefile), passphrase);    ts.load(new java.io.fileinputstream(truststorefile), passphrase);     keymanagerfactory kmf = keymanagerfactory.getinstance("sunx509");    kmf.init(ks, passphrase);     trustmanagerfactory tmf = trustmanagerfactory.getinstance("sunx509");    tmf.init(ts);     sslcontext sslc = sslcontext.getinstance("tls");         sslc.init(kmf.getkeymanagers(), tmf.gettrustmanagers(), null);      sslengine serverengine = sslc.createsslengine();    serverengine.setuseclientmode(false);    serverengine.setenablesessioncreation(true);    serverengine.setwantclientauth(true); 

once sslengine instantiated, process inbound data using unwrap/wrap methods using code straight out of official jsse samples:

   log("----");     serverresult = serverengine.unwrap(innetdata, inappdata);    log("server unwrap: ", serverresult);    rundelegatedtasks(serverresult, serverengine);     log("----");     serverresult = serverengine.wrap(outappdata, outnetdata);    log("server wrap: ", serverresult);    rundelegatedtasks(serverresult, serverengine); 

the first part of handshake seems work fine. client sends handshake message , server responds message 4 records:

handshake (22) - server_hello (2)  - certificate (11)  - server_key_exchange (12) - certificate_request (13)  - server_hello_done (14) 

next, client sends message 3 parts:

handshake (22)  - certificate (11)  - client_key_exchange (16)  change_cipher_spec (20)  - client_hello (1)  handshake (22)  *** encrypted message **** 

the sslengine unwraps client request , parses records wrap method produces 0 bytes handshake status of ok/need_unwrap. in other words, there's nothing me send client , handshake comes screeching halt.

this stuck.

in debugger, can see sslengine, serverhandshaker, doesn't find peer certs. rather obvious when @ certificate record client 0 bytes long. why?

i can assume there's wrong helloserver response can't seem put finger on it. server seems sending valid cert client isn't sending back. there problem keystore? or truststore? or have way i'm instantiating sslengine? i'm stumped.

couple other points:

• the keystore , truststore referenced in code snippit above created using following tutorial: http://www.techbrainwave.com/?p=953
• i'm using firefox 10 , ie 9 client test server. same results both web clients.
• i'm using sun/oracle jdk 6 , java secure socket extension (jsse) comes bundled it.

i forward guidance might have please don't tell me i'm nuts or use netty or grizzly or other existing solution. not option @ time. want understand i'm doing wrong.

thanks in advance!

you got need_unwrap, unwrap. in turn might give buffer_underflow, means have read , retry unwrap.

similarly when need_wrap, wrap: in turn might give buffer_overflow, means have write , retry wrap.

that wrap or unwrap might in turn might tell operation: wrap or unwrap.

just tells do.