메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


./spark-sql 실행시 mysql driver jar파일도 classpath에 등록했는데도 아래오 같은 오류가 발생시

hive-site.xml의 아래 항목을 찾아서 <value>0s</value>를 <value>0</value>로 변경해준다.

 <property>

    <name>hive.metastore.client.socket.lifetime</name>

    <value>0s</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      MetaStore Client socket lifetime in seconds. After this time is exceeded, client

      reconnects on the next MetaStore operation. A value of 0s means the connection

      has an infinite lifetime.

    </description>

  </property>

--->

 <property>

    <name>hive.metastore.event.clean.freq</name>

    <value>0</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      Frequency at which timer task runs to purge expired events in metastore.

    </description>

  </property>

  <property>

    <name>hive.metastore.event.expiry.duration</name>

    <value>0</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      Duration after which events expire from events table

    </description>

  </property>




---------------------------------오류내용--------------------------------------

16/06/09 13:37:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/06/09 13:37:46 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

16/06/09 13:37:46 INFO ObjectStore: ObjectStore, initialize called

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateColumns unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateConstraints unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.autoCreateAll unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateTables unknown - will be ignored

16/06/09 13:37:46 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/06/09 13:37:46 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/06/09 13:37:46 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing

16/06/09 13:37:47 INFO ObjectStore: Initialized ObjectStore

16/06/09 13:37:47 INFO HiveMetaStore: Added admin role in metastore

16/06/09 13:37:47 INFO HiveMetaStore: Added public role in metastore

16/06/09 13:37:47 INFO HiveMetaStore: No user is added in admin role, since config is empty

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:101)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)

        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)

        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)

        ... 11 more

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)

        ... 16 more

Caused by: java.lang.NumberFormatException: For input string: "0s"

        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

        at java.lang.Long.parseLong(Long.java:589)

        at java.lang.Long.parseLong(Long.java:631)

        at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1187)

        at org.apache.hadoop.hive.conf.HiveConf.getLongVar(HiveConf.java:1229)

        at org.apache.hadoop.hive.conf.HiveConf.getLongVar(HiveConf.java:1242)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:429)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)

        at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)

        ... 21 more

번호 제목 날짜 조회 수
145 hue.axes_accessattempt테이블의 username컬럼에 NULL 혹은 space가 들어갈수도 있음. 2021.11.03 90865
144 Hive Query Examples from test code (2 of 2) 2014.03.26 13505
143 insert hbase by hive ... error occured after 5 hours..HMaster가 뜨지 않는 장애에 대한 복구 방법 2014.04.29 9666
142 hive 2.0.1 설치및 mariadb로 metastore 설정 2016.06.03 9063
141 spark-sql실행시 The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH오류 발생시 조치사항 2016.06.09 7887
140 Spark에서 Serializable관련 오류및 조치사항 2017.04.21 7743
139 oracle to hive data type정리표 2018.08.22 7375
» spark-sql실행시 Caused by: java.lang.NumberFormatException: For input string: "0s" 오류발생시 조치사항 2016.06.09 7108
137 [impala]insert into db명.table명 select a, b from db명.table명 쿼리 수행시 "Memory limit exceeded: Failed to allocate memory for Parquet page index"오류 조치 방법 2023.05.31 6326
136 java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from...오류해결방법 2015.06.16 5943
135 [CDP7.1.7]impala-shell을 이용하여 kudu table에 insert/update수행시 발생하는 오류(Transport endpoint is not connected (error 107)) 발생시 확인할 내용 2023.11.30 5875
134 의사분산모드에서 presto설치하기 2014.03.31 5515
133 Hive 사용법 및 쿼리 샘플코드 2013.03.07 5441
132 upsert구현방법(년-월-일 파티션을 기준으로) 및 테스트 script file 2018.07.03 5348
131 [CDP7.1.7]impala-shell수행시 간헐적으로 "-k requires a valid kerberos ticket but no valid kerberos ticket found." 오류 2023.11.16 5307
130 Hive Query Examples from test code (1 of 2) 2014.03.26 5245
129 [Kudu]ERROR: Unable to advance iterator for node with id '2' for Kudu table 'impala::core.pm0_abdasubjct': Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) 2023.03.16 5210
128 Hive+mysql 설치 및 환경구축하기 file 2013.03.07 5120
127 hive job실행시 meta정보를 원격의 mysql에 저장하는 경우 설정방법 2014.05.28 5098
126 kudu rebalance수행 command예시 2022.01.17 5072
위로