메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


hive-site.xml에서 아래 3개의 <value>에서 ms를 뺀다.


 <property>

    <name>hive.hmshandler.retry.interval</name>

    <value>2000ms</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.

      The time between HMSHandler retry attempts on failure.

    </description>

  </property>


  <property>

    <name>hive.llap.am.liveness.connection.sleep.between.retries.ms</name>

    <value>2000ms</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.

      Sleep duration while waiting to retry connection failures to the AM from the daemon for

      the general keep-alive thread (milliseconds).

    </description>

  </property>



  <property>

    <name>hive.llap.task.communicator.connection.sleep.between.retries.ms</name>

    <value>2000ms</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.

      Sleep duration (in milliseconds) to wait before retrying on error when obtaining a

      connection to LLAP daemon from Tez AM.

    </description>

  </property>




---------------------------------오류내용-----------------------------------------

16/06/09 13:57:38 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0

SET spark.sql.hive.version=0.13.1

16/06/09 13:57:39 ERROR log: Got exception: java.lang.NumberFormatException For input string: "2000ms"

java.lang.NumberFormatException: For input string: "2000ms"

        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

        at java.lang.Integer.parseInt(Integer.java:580)

        at java.lang.Integer.parseInt(Integer.java:615)

        at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1134)

        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:87)

        at com.sun.proxy.$Proxy6.get_all_databases(Unknown Source)

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:837)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)

        at com.sun.proxy.$Proxy7.getAllDatabases(Unknown Source)

        at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1098)

        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:671)

        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:662)

        at org.apache.hadoop.hive.cli.CliDriver.getCommandCompletor(CliDriver.java:540)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:168)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

16/06/09 13:57:39 ERROR log: Converting exception to MetaException

16/06/09 13:57:39 ERROR FunctionRegistry: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.lang.NumberFormatException For input string: "2000ms")

spark-sql> 16/06/09 13:57:39 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@sda2:47136/user/Executor#-853352827] with ID 2

16/06/09 13:57:39 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@so2:35213/user/Executor#-894168911] with ID 0

16/06/09 13:57:39 INFO BlockManagerMasterActor: Registering block manager sda2:58779 with 265.1 MB RAM, BlockManagerId(2, sda2, 58779)

16/06/09 13:57:39 INFO BlockManagerMasterActor: Registering block manager so2:33469 with 265.1 MB RAM, BlockManagerId(0, so2, 33469)

16/06/09 13:57:41 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@so-db2:35159/user/Executor#-473750792] with ID 1

16/06/09 13:57:41 INFO BlockManagerMasterActor: Registering block manager so-db2:47990 with 265.1 MB RAM, BlockManagerId(1, so-db2, 47990)

번호 제목 날짜 조회 수
41 spark 온라인 책자링크 (제목 : mastering-apache-spark) 2016.05.25 164
40 파일끝에 붙는 ^M 일괄 지우기(linux, unix(AIX)) 혹은 파일내에 있는 ^M지우기 2016.09.24 168
39 spark에서 hive table을 읽어 출력하는 예제 소스 2017.03.09 179
38 참고할만한 spark예제를 설명하는 사이트 2016.11.11 185
37 JavaStreamingContext를 이용하여 스트림으로 들어오는 문자열 카운트 소스 2017.03.30 210
36 How-to: Tune Your Apache Spark Jobs (Part 2) file 2016.10.31 221
35 ./spark-sql 실행시 "java.lang.NumberFormatException: For input string: "1s"오류발생시 조치사항 2016.06.09 226
34 Spark Streaming으로 유실 없는 스트림 처리 인프라 구축하기 2016.03.11 262
33 spark 2.0.0의 api를 이용하는 예제 프로그램 2017.03.15 274
32 spark2.0.0에서 hive 2.0.1 table을 읽어 출력하는 예제 소스(HiveContext, SparkSession, SQLContext) 2017.03.09 283
31 scala application 샘플소스(SparkSession이용) 2018.03.07 296
» spark-sql실행시 ERROR log: Got exception: java.lang.NumberFormatException For input string: "2000ms" 오류발생시 조치사항 2016.06.09 299
29 spark 2.0.0를 windows에서 실행시 로컬 파일을 읽을때 발생하는 오류 해결 방법 2017.01.12 311
28 start-all.sh로 spark데몬 기동시 "JAVA_HOME is not set"오류 발생시 조치사항 2016.08.01 317
27 Could not compute split, block input-0-1517397051800 not found형태의 오류가 발생시 조치방법 2018.02.01 333
26 spark-submit으로 spark application실행하는 다양한 방법 2016.05.25 387
25 Spark에서 KafkaUtils.createStream()를 이용하여 이용하여 kafka topic에 접근하여 객채로 저장된 값을 가져오고 처리하는 예제 소스 2017.04.26 393
24 spark notebook 0.7.0설치및 설정 2016.11.14 464
23 Scala를 이용한 Streaming예제 2018.03.08 501
22 Scala버젼 변경 혹은 상황에 맞게 Spark소스 컴파일하기 2016.05.31 504
위로