메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


./spark-sql 실행시 mysql driver jar파일도 classpath에 등록했는데도 아래오 같은 오류가 발생시

hive-site.xml의 아래 항목을 찾아서 <value>0s</value>를 <value>0</value>로 변경해준다.

 <property>

    <name>hive.metastore.client.socket.lifetime</name>

    <value>0s</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      MetaStore Client socket lifetime in seconds. After this time is exceeded, client

      reconnects on the next MetaStore operation. A value of 0s means the connection

      has an infinite lifetime.

    </description>

  </property>

--->

 <property>

    <name>hive.metastore.event.clean.freq</name>

    <value>0</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      Frequency at which timer task runs to purge expired events in metastore.

    </description>

  </property>

  <property>

    <name>hive.metastore.event.expiry.duration</name>

    <value>0</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      Duration after which events expire from events table

    </description>

  </property>




---------------------------------오류내용--------------------------------------

16/06/09 13:37:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/06/09 13:37:46 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

16/06/09 13:37:46 INFO ObjectStore: ObjectStore, initialize called

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateColumns unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateConstraints unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.autoCreateAll unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateTables unknown - will be ignored

16/06/09 13:37:46 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/06/09 13:37:46 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/06/09 13:37:46 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing

16/06/09 13:37:47 INFO ObjectStore: Initialized ObjectStore

16/06/09 13:37:47 INFO HiveMetaStore: Added admin role in metastore

16/06/09 13:37:47 INFO HiveMetaStore: Added public role in metastore

16/06/09 13:37:47 INFO HiveMetaStore: No user is added in admin role, since config is empty

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:101)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)

        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)

        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)

        ... 11 more

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)

        ... 16 more

Caused by: java.lang.NumberFormatException: For input string: "0s"

        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

        at java.lang.Long.parseLong(Long.java:589)

        at java.lang.Long.parseLong(Long.java:631)

        at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1187)

        at org.apache.hadoop.hive.conf.HiveConf.getLongVar(HiveConf.java:1229)

        at org.apache.hadoop.hive.conf.HiveConf.getLongVar(HiveConf.java:1242)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:429)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)

        at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)

        ... 21 more

번호 제목 날짜 조회 수
427 import 혹은 export할때 hive파일의 default 구분자는 --input-fields-terminated-by "x01"와 같이 지정해야함 2014.05.20 6700
426 Last transaction was partial에 따른 Unable to load database on disk오류 발생시 조치사항 2018.08.03 6652
425 hadoop 2.6.0 기동(에코시스템 포함)및 wordcount 어플리케이션을 이용한 테스트 2015.05.05 6516
424 다수의 로그 에이전트로 부터 로그를 받아 각각의 파일로 저장하는 방법(interceptor및 multiplexing) 2014.04.04 6439
423 hadoop및 ecosystem에서 사용되는 명령문 정리 2014.05.28 6434
422 [impala]insert into db명.table명 select a, b from db명.table명 쿼리 수행시 "Memory limit exceeded: Failed to allocate memory for Parquet page index"오류 조치 방법 2023.05.31 6326
421 특정파일이 생성되어야 action이 실행되는 oozie job만들기(coordinator.xml) 2014.05.20 6259
420 Hadoop Cluster 설치 (Hadoop+Zookeeper+Hbase) file 2013.03.07 6097
419 HBase 설치하기 – Fully-distributed 2013.03.12 6012
418 java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from...오류해결방법 2015.06.16 5943
417 Hbase Shell 명령 정리 2013.04.01 5924
416 [CDP7.1.7]impala-shell을 이용하여 kudu table에 insert/update수행시 발생하는 오류(Transport endpoint is not connected (error 107)) 발생시 확인할 내용 2023.11.30 5871
415 HBASE Client API : 기본 기능 정리 file 2013.04.01 5851
414 "java.net.NoRouteToHostException: 호스트로 갈 루트가 없음" 오류시 확인및 조치할 사항 2016.04.01 5809
413 hbase shell 필드 검색 방법 2015.05.24 5694
412 JobHistory 서버 기동시 HDFS상에 특정 폴더를 생성할 수 없어서 기동하지 못하는 경우 조치 2018.05.29 5651
411 이클립스에서 생성한 jar 파일 hadoop 으로 실행하기 file 2013.03.06 5536
410 의사분산모드에서 presto설치하기 2014.03.31 5515
409 AIX 7.1에 MariaDB 10.2 소스 설치 2016.09.24 5451
408 sqoop 1.4.4 설치및 테스트 2014.04.21 5445
위로