메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


./spark-sql 실행시 mysql driver jar파일도 classpath에 등록했는데도 아래오 같은 오류가 발생시

hive-site.xml의 아래 항목을 찾아서 <value>0s</value>를 <value>0</value>로 변경해준다.

 <property>

    <name>hive.metastore.client.socket.lifetime</name>

    <value>0s</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      MetaStore Client socket lifetime in seconds. After this time is exceeded, client

      reconnects on the next MetaStore operation. A value of 0s means the connection

      has an infinite lifetime.

    </description>

  </property>

--->

 <property>

    <name>hive.metastore.event.clean.freq</name>

    <value>0</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      Frequency at which timer task runs to purge expired events in metastore.

    </description>

  </property>

  <property>

    <name>hive.metastore.event.expiry.duration</name>

    <value>0</value>

    <description>

      Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.

      Duration after which events expire from events table

    </description>

  </property>




---------------------------------오류내용--------------------------------------

16/06/09 13:37:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/06/09 13:37:46 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

16/06/09 13:37:46 INFO ObjectStore: ObjectStore, initialize called

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateColumns unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateConstraints unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.autoCreateAll unknown - will be ignored

16/06/09 13:37:46 INFO Persistence: Property datanucleus.schema.validateTables unknown - will be ignored

16/06/09 13:37:46 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/06/09 13:37:46 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/06/09 13:37:46 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/06/09 13:37:47 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing

16/06/09 13:37:47 INFO ObjectStore: Initialized ObjectStore

16/06/09 13:37:47 INFO HiveMetaStore: Added admin role in metastore

16/06/09 13:37:47 INFO HiveMetaStore: Added public role in metastore

16/06/09 13:37:47 INFO HiveMetaStore: No user is added in admin role, since config is empty

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:101)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)

        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)

        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)

        ... 11 more

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)

        ... 16 more

Caused by: java.lang.NumberFormatException: For input string: "0s"

        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

        at java.lang.Long.parseLong(Long.java:589)

        at java.lang.Long.parseLong(Long.java:631)

        at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1187)

        at org.apache.hadoop.hive.conf.HiveConf.getLongVar(HiveConf.java:1229)

        at org.apache.hadoop.hive.conf.HiveConf.getLongVar(HiveConf.java:1242)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:429)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)

        at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)

        ... 21 more

번호 제목 날짜 조회 수
250 kafkaWordCount.scala의 producer와 consumer 클래스를 이용하여 kafka를 이용한 word count 테스트 하기 2016.08.02 3561
249 프로세스를 확인해서 프로세스를 삭제하는 shell script예제(cryptonight) 2018.02.02 3545
248 [Impala jdbc]CDP7.1.7환경에서 java프로그램을 이용하여 kerberized impala cluster에 접근하여 SQL을 수행하는 방법 2023.08.22 3544
247 eclipse editor 설정방법 2022.02.01 3542
246 DBCP Datasource(org.apache.commons.dbcp.BasicDataSource) 설정 및 속성 설명 2016.09.26 3538
245 Cloudera Manager 5.x설치시 embedded postgresql를 사용하는 경우의 관리정보 2018.04.13 3537
244 spark stream처리할때 두개의 client프로그램이 동일한 checkpoint로 접근할때 발생하는 오류 내용 2018.01.16 3537
243 oozie db변경후 재기동시 "Table 'oozie.VALIDATE_CONN' doesn't exist" 오류 발생시 조치방법 2018.05.23 3536
242 워킹 메모리에 대한 정보를 처리하는 클래스 파일 2016.07.21 3533
241 --master yarn 옵션으로 spark client프로그램 실행할때 메모리 부족 오류발생시 조치방법 2016.05.27 3522
240 원보드 컴퓨터 비교표 file 2014.08.04 3521
239 RDF4J의 rdf4j-server.war가 제공하는 RESTFul API를 이용한 CRUD테스트(트랜잭션처리) 2017.08.30 3517
238 [Ranger]RangerAdminRESTClient Error gertting pplicies; Received NULL response!!, secureMode=true, user=rangerkms/node01.gooper.com@ GOOPER.COM (auth:KERBEROS), serviceName=cm_kms 2023.06.27 3516
237 컴퓨터 무한 재부팅 원인및 조치방법 file 2017.12.05 3512
236 [TLS]pkcs12형식의 인증서 생성및 jks형식 인증서 생성 커맨드 예시 2022.03.15 3511
235 [postgresql 9.x] PostgreSQL Replication 구축하기 2018.07.17 3510
234 halyard 1.3을 다른 서버로 이전하는 방법 2017.07.05 3500
233 [vi]블럭 및 문서내 복사등에 관련된 명령어 2017.02.17 3497
232 [Dovecot] -ERR [SYS/PERM] Permission denied 2017.06.13 3493
231 VPS에서는 root로 실행해도 swap파일을 만들지 못하게 만들어 두었지만 swap파일을 생성하는 방법 2017.06.20 3474
위로