java - What could cause socket ConnectException: Connection timed out? -

we have webstart client communicates server sending serialized objects on https using java.net.httpsurlconnection.

everything works fine on local machine , on test servers located in our office, i'm experiencing very, strange issue occurring on our production , staging servers (and sporadically @ that). main difference know of between servers , ones in our office located elsewhere , client-server communication them considerably slower, worked fine long time in production prior well.

anyway, here's what's happening:

• the client, after setting options such read timeout , properties such content-type on httpurlconnection, calls getoutputstream() on stream write to.
• at point, can tell, client hangs period of time.
• the client throws following exception:

 java.net.connectexception: connection timed out: connect     @ java.net.plainsocketimpl.socketconnect(native method)     @ java.net.plainsocketimpl.doconnect(unknown source)     @ java.net.plainsocketimpl.connecttoaddress(unknown source)     @ java.net.plainsocketimpl.connect(unknown source)     @ java.net.sockssocketimpl.connect(unknown source)     @ java.net.socket.connect(unknown source)     @ com.sun.net.ssl.internal.ssl.sslsocketimpl.connect(unknown source)     @ com.sun.net.ssl.internal.ssl.basesslsocketimpl.connect(unknown source)     @ sun.net.networkclient.doconnect(unknown source)     @ sun.net.www.http.httpclient.openserver(unknown source)     @ sun.net.www.http.httpclient.openserver(unknown source)     @ sun.net.www.protocol.https.httpsclient.(unknown source)     @ sun.net.www.protocol.https.httpsclient.new(unknown source)     @ sun.net.www.protocol.https.abstractdelegatehttpsurlconnection.getnewhttpclient(unknown source)     @ sun.net.www.protocol.http.httpurlconnection.plainconnect(unknown source)     @ sun.net.www.protocol.https.abstractdelegatehttpsurlconnection.connect(unknown source)     @ sun.net.www.protocol.http.httpurlconnection.getoutputstream(unknown source)     @ sun.net.www.protocol.https.httpsurlconnectionimpl.getoutputstream(unknown source) 

note not sockettimeoutexception, connect() method on httpurlconnection says throws if timeout expires before connection can established. also, when happens able call conn.getresponsecode() , response code of 200.

• on server side, eofexception thrown in objectinputstream's constructor, tries read serialization header fails because client never gets outputstream write to.

in case helps, here calls being made on httpsurlconnection prior call getoutputstream() (edited show calls being made rather whole structure of code doing this):

httpsurlconnection conn = (httpsurlconnection) url.openconnection(); conn.setusecaches(false); conn.setreadtimeout(30000); conn.setrequestproperty("cookie", cookie); conn.setdooutput(true); conn.setrequestproperty("content-type", "application/x-java-serialized-object"); conn.getoutputstream(); 

the thing is, have no idea how of happening, given happens occasionally (no clear pattern of activity can tell) , when there's (relatively) high latency between client , server.

given i've been able find far java.net.connectexception: connect timed out, wondered if weren't network or firewall issue on network our servers running on... doesn't make sense me given request getting through servlet. also, other apps running on same network have not reported similar issues.

does have idea cause of be, or should investigate?

we have come across these in similar case yours. @ high load , not easy reproduce on test. have not fixed yet steps went through.

if it's firewall issue, connection refused or sockettimeout exception.

1) able track these requests in access log on server - show http status 200 or 404 or else? in our case, server (iis in case) logs showed client closed connection , not server. mystery.

update: if client gets 200, server has sent response suspect response byte-size (if recorded in access logs) will show different value of normal response size request.

if shows same size of response, have (may not plausible) condition server actually responded correctly client did not response because connection terminated somewhere in between.

2) network admin teams looked @ tcp/ip traffic determine end (or intermediate router) terminating http / tcp-ip conversation. , once understand end terminating connection @ why. knowledgable enough run snoop

3) there max number of requests configured/restricted on server - , throttling connections?

4) there intermediate load balancers @ requests dropped?

update: 1 more thing wanted to, did not complete create static route between client , server reduce number of hops in between , ensure no network related connection drops. see http://en.wikipedia.org/wiki/static_routing

5) suggestion setting connecttimeout see if these work higher value. update: might want try conn.geterrorstream()

returns error stream if connection failed server sent useful data nonetheless. if connection not connected, or if server did not have error while connecting or if server had error no error data sent, method return null.

6) try taking set of thread dumps on server 5 seconds apart, see if thread shows these incoming requests on server.

update: of today learnt live problem, because totalled failure rate 200-300 out of 400,000 requests per day 0.00075 %