Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다.
문의사항은 gooper@gooper.com로 메일을
보내주세요.
spark spark-sql실행시 ERROR log: Got exception: java.lang.NumberFormatException For input string: "2000ms" 오류발생시 조치사항
hive-site.xml에서 아래 3개의 <value>에서 ms를 뺀다.
<property>
<name>hive.hmshandler.retry.interval</name>
<value>2000ms</value>
<description>
Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.
The time between HMSHandler retry attempts on failure.
</description>
</property>
<property>
<name>hive.llap.am.liveness.connection.sleep.between.retries.ms</name>
<value>2000ms</value>
<description>
Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.
Sleep duration while waiting to retry connection failures to the AM from the daemon for
the general keep-alive thread (milliseconds).
</description>
</property>
<property>
<name>hive.llap.task.communicator.connection.sleep.between.retries.ms</name>
<value>2000ms</value>
<description>
Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.
Sleep duration (in milliseconds) to wait before retrying on error when obtaining a
connection to LLAP daemon from Tez AM.
</description>
</property>
---------------------------------오류내용-----------------------------------------
16/06/09 13:57:38 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
SET spark.sql.hive.version=0.13.1
16/06/09 13:57:39 ERROR log: Got exception: java.lang.NumberFormatException For input string: "2000ms"
java.lang.NumberFormatException: For input string: "2000ms"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1134)
at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:87)
at com.sun.proxy.$Proxy6.get_all_databases(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy7.getAllDatabases(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1098)
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:671)
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:662)
at org.apache.hadoop.hive.cli.CliDriver.getCommandCompletor(CliDriver.java:540)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:168)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/06/09 13:57:39 ERROR log: Converting exception to MetaException
16/06/09 13:57:39 ERROR FunctionRegistry: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.lang.NumberFormatException For input string: "2000ms")
spark-sql> 16/06/09 13:57:39 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@sda2:47136/user/Executor#-853352827] with ID 2
16/06/09 13:57:39 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@so2:35213/user/Executor#-894168911] with ID 0
16/06/09 13:57:39 INFO BlockManagerMasterActor: Registering block manager sda2:58779 with 265.1 MB RAM, BlockManagerId(2, sda2, 58779)
16/06/09 13:57:39 INFO BlockManagerMasterActor: Registering block manager so2:33469 with 265.1 MB RAM, BlockManagerId(0, so2, 33469)
16/06/09 13:57:41 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@so-db2:35159/user/Executor#-473750792] with ID 1
16/06/09 13:57:41 INFO BlockManagerMasterActor: Registering block manager so-db2:47990 with 265.1 MB RAM, BlockManagerId(1, so-db2, 47990)