메뉴 건너뛰기

검색창 닫기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


fuseki/jena/ 동시에 많은 요청이 endpoint로 몰려서java.net.NoRouteToHostException가 발생하는 경우의 처리방법

총관리자 2016.10.17 14:27 조회 수 : 664

오류내용#1처럼 fuseki endpoint를 통하여 요청을 보낼때 순간적으로 많은 요청이 발생하면 많은양의 소켓케넥션이 TIME-WAIT상태(netstat -a | grep 사용포트(예,23030))가 되면서 오류가 발생할 수 있는데 이럴때는

echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
and/or
echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
를 수행하여 TIME-WAIT상태의 소켓 커넥션을 재사용할 수 있도록 설정해준다.

이렇게 처리해주면 오류내용#2가 발생하는데 이는 커넥션 재사용에 커넥션은 형성되었는데 이를 jena가 처리하지 못하는 과부하 상태를 의미한다.

* 참고 : http://stackoverflow.com/questions/1572215/how-to-avoid-a-noroutetohostexception
                                                                                             
-------------------------------오류내용#1-------------------------------------------
[2016-10-12 13:56:57,031] [service.llluminationStateJobService] [execute(#118)] [DEBUG] Exception  ....................................> HttpException: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다: Unexpected error making the query: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다 
[2016-10-12 13:56:57,031] [service.LectureTimeOnStateJobService] [execute(#112)] [DEBUG] Exception  ....................................> HttpException: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다: Unexpected error making the query: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다 
[2016-10-12 13:56:57,031] [service.ElectricPowerStateJobService] [execute(#115)] [DEBUG] Exception  ....................................> HttpException: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다: Unexpected error making the query: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다 
[2016-10-12 13:56:57,032] [service.HumanExistStateJobService] [execute(#116)] [DEBUG] Exception  ....................................> HttpException: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다: Unexpected error making the query: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다 
[2016-10-12 13:56:57,032] [service.ClimateStateJobService] [execute(#115)] [DEBUG] Exception  ....................................> HttpException: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다: Unexpected error making the query: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다 
[2016-10-12 13:56:57,034] [service.ProjectorStateJobService] [execute(#115)] [DEBUG] Exception  ....................................> HttpException: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다: Unexpected error making the query: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다 
[2016-10-12 13:56:57,034] [service.LectureTimeOffStateJobService] [execute(#112)] [DEBUG] Exception  ....................................> HttpException: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다: Unexpected error making the query: java.net.NoRouteToHostException: 요청한 주소를 배정할 수 없습니다 



-------------------------------오류내용#2-----------------------------------------
[2016-10-13 10:44:24,487] [sf.SparqlService] [updateSparql(#376)] [DEBUG] delete->insert sparql start............................
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#138)] [DEBUG] count of idxVals : 2
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#139)] [DEBUG] values of idxVals : [http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data, http://www.iotoasis.org/herit-in/herit-cse/TempSens
or_LR0001TS0005/status/Data/CONTENT_INST_2409764]
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#146)] [DEBUG] sparql to make ===========>
 prefix o: <http://www.iotoasis.org/ontology/>  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>  delete  { <@{arg0}> o:hasLatestContentInstance ?o . }  WHERE   { <@{arg0}> o:hasLatestContentInstance  ?o  .}
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#268)] [DEBUG] sparql made ===========>
 prefix o: <http://www.iotoasis.org/ontology/>  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>  delete  { <http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data> o:hasLatestContentInstance
 ?o . }  WHERE   { <http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data> o:hasLatestContentInstance  ?o  .}
org.apache.jena.atlas.web.HttpException: java.net.ConnectException: 연결이 거부됨
        at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1202)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:803)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:578)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:534)
        at org.apache.jena.sparql.modify.UpdateProcessRemote.execute(UpdateProcessRemote.java:79)
        at com.gooper.icbms.sda.sf.SparqlService.runModifySparql(SparqlService.java:404)
        at com.gooper.icbms.sda.sf.SparqlService.updateSparql(SparqlService.java:378)
        at com.gooper.icbms.sda.sf.TripleService.addLatestContentInstance(TripleService.java:173)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MDataSubscribe$1.run(AvroOneM2MDataSubscribe.java:116)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: 연결이 거부됨
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:120)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1186)
                                                                                                                                                                                                              6182859,25-32 99%
[2016-10-13 10:44:24,487] [sf.TripleService] [addLatestContentInstance(#172)] [DEBUG] this.getInstanceUri() =====>http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data/C
ONTENT_INST_2409764
[2016-10-13 10:44:24,487] [sf.SparqlService] [updateSparql(#376)] [DEBUG] delete->insert sparql start............................
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#138)] [DEBUG] count of idxVals : 2
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#139)] [DEBUG] values of idxVals : [http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data, http://www.iotoasis.o
rg/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data/CONTENT_INST_2409764]
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#146)] [DEBUG] sparql to make ===========>
 prefix o: <http://www.iotoasis.org/ontology/>  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>  delete  { <@{arg0}> o:hasLatestContentInstance ?o . }  WHERE   { <@{arg0}> o:hasLatestC
ontentInstance  ?o  .}
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#268)] [DEBUG] sparql made ===========>
 prefix o: <http://www.iotoasis.org/ontology/>  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>  delete  { <http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Da
ta> o:hasLatestContentInstance ?o . }  WHERE   { <http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data> o:hasLatestContentInstance  ?o  .}
org.apache.jena.atlas.web.HttpException: java.net.ConnectException: 연결이 거부됨
        at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1202)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:803)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:578)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:534)
        at org.apache.jena.sparql.modify.UpdateProcessRemote.execute(UpdateProcessRemote.java:79)
        at com.gooper.icbms.sda.sf.SparqlService.runModifySparql(SparqlService.java:404)
        at com.gooper.icbms.sda.sf.SparqlService.updateSparql(SparqlService.java:378)
        at com.gooper.icbms.sda.sf.TripleService.addLatestContentInstance(TripleService.java:173)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MDataSubscribe$1.run(AvroOneM2MDataSubscribe.java:116)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: 연결이 거부됨
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:120)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
                                                                                                                                                                                6182859,25-32 99%
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:120)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1186)
        ... 11 more
[2016-10-13 10:44:24,470] [sf.SparqlService] [runModifySparql(#407)] [DEBUG] Exception message in runModifySparql() =====> java.net.ConnectException: 연결이 거부됨
[2016-10-13 10:44:24,470] [util.Utils] [restartFuseki(#494)] [DEBUG] fuseki-server restart start ...................
[2016-10-13 10:44:24,470] [util.Utils] [killFuseki(#416)] [DEBUG] killFuseki start==========================>
[2016-10-13 10:44:24,487] [onem2m.AvroOneM2MDataSubscribe$1] [run(#95)] [DEBUG] count data from kafka broker : 39995
[2016-10-13 10:44:24,487] [sf.TripleService] [getTriple(#73)] [DEBUG] === ri : CONTENT_INST_2409764,  ty : 4 ====>include
contentType : normal
[2016-10-13 10:44:24,487] [onem2m.AvroOneM2MDataSubscribe$1] [run(#115)] [DEBUG] delete->insert o:hasLatestContentInstance start....................
[2016-10-13 10:44:24,487] [sf.TripleService] [addLatestContentInstance(#171)] [DEBUG] this.getParentResourceUri() ====> http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data
[2016-10-13 10:44:24,487] [sf.TripleService] [addLatestContentInstance(#172)] [DEBUG] this.getInstanceUri() =====>http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data/CONTENT_INST_2409764
[2016-10-13 10:44:24,487] [sf.SparqlService] [updateSparql(#376)] [DEBUG] delete->insert sparql start............................
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#138)] [DEBUG] count of idxVals : 2
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#139)] [DEBUG] values of idxVals : [http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data, http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data/CONT
ENT_INST_2409764]
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#146)] [DEBUG] sparql to make ===========>
 prefix o: <http://www.iotoasis.org/ontology/>  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>  delete  { <@{arg0}> o:hasLatestContentInstance ?o . }  WHERE   { <@{arg0}> o:hasLatestContentInstance  ?o  .}
[2016-10-13 10:44:24,487] [sf.SparqlService] [makeSparql(#268)] [DEBUG] sparql made ===========>
 prefix o: <http://www.iotoasis.org/ontology/>  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>  delete  { <http://www.iotoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data> o:hasLatestContentInstance ?o . }  WHERE   { <http://www.i
otoasis.org/herit-in/herit-cse/TempSensor_LR0001TS0005/status/Data> o:hasLatestContentInstance  ?o  .}
org.apache.jena.atlas.web.HttpException: java.net.ConnectException: 연결이 거부됨
        at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1202)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:803)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:578)
        at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:534)
        at org.apache.jena.sparql.modify.UpdateProcessRemote.execute(UpdateProcessRemote.java:79)
        at com.gooper.icbms.sda.sf.SparqlService.runModifySparql(SparqlService.java:404)
        at com.gooper.icbms.sda.sf.SparqlService.updateSparql(SparqlService.java:378)
        at com.gooper.icbms.sda.sf.TripleService.addLatestContentInstance(TripleService.java:173)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MDataSubscribe$1.run(AvroOneM2MDataSubscribe.java:116)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: 연결이 거부됨
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:120)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1186)
        ... 11 more
이 게시물을
이 글의 추천인 목록 목록
번호 제목 날짜 조회 수
58 update를 많이 하면 heap memory가 많이 소진되고 최종적으로 OOM가 발생하는데 이에 대한 설명 2017.04.10 818
57 LUBM 개수별 hadoop HDFS data사이즈 정리 2017.04.06 214
56 protege 설명및 사용법 file 2017.04.04 3025
55 fuseki가 제공하는 web ui를 통해서 dataset를 remove->create할 경우 동일한 동일한 이름으로 지정했을때 fuseki-server.jar가 뜨지 않는 현상 2017.02.03 495
54 https://github.com/Merck/Halyard프로젝트 컴파일및 배포/테스트 2017.01.24 185
53 federated query 예제 2017.01.19 324
52 Halyard - RDF4J와 Apache HBase를 이용하여 구현된 TripleStore이며 SPARQL 1.1쿼리를 지원한다. 2016.12.29 734
51 jena의 data폴더를 hadoop nfs를 이용하여 HDFS상의 폴더에 마운트 시키고 fuseki를 통하여 inert를 시도했을때 transaction 오류 발생 2016.12.02 174
50 Jena 2.3를 Hadoop 2.7.2의 NFS로 mount하고 fuseki를 이용하여 start할때 오류 메세지 2016.12.02 1644
49 S2RDF를 이용한 다른 버젼의 github링크 2016.12.02 274
48 데이타 분석및 머신러닝에 도움이 도움이 되는 사이트 2016.11.04 455
47 How-to: Build a Complex Event Processing App on Apache Spark and Drools file 2016.10.31 360
» 동시에 많은 요청이 endpoint로 몰려서java.net.NoRouteToHostException가 발생하는 경우의 처리방법 2016.10.17 664
45 프로그래밍 언어별 딥러닝 라이브러리 정리 file 2016.10.05 314
44 모두를 위한 머신러닝과 딥러닝의 강의 file 2016.09.27 418
43 특정문자열이나 URI를 임의로 select 절에 지정하여 사용할때 사용하는 sparql 문장 2016.08.25 457
42 jena jar파일실행시 org.apache.jena.tdb.TDB.init에서 java.lang.NullPointerException발생시 조치사항 2016.08.19 412
41 Jena는 기본적으로 multi thread환경을 지원하지 않는다. 2016.08.16 322
40 [Elephas] Jena Elephas를 이용하여 Spark에서 rdfTriples의 RDD를 만들고 RDD관련 작업하는 샘플소스 2016.08.10 203
39 슬라이딩 윈도우 예제 2016.07.28 191
쓰기 태그
위로