bin/spark-shell 실행시 아래와 같은 오류가 발생할 수 있는데 두가지 가능성이 있다.

(Spark가 spark-shell을 실행하면서 Hive를 지원하기 위한 설정을 하면서 Hive의 metastore를 사용할 수 없어서(예, Hive가 설치되지 않거나 했을때) derby를 이용하여 Hive의 metastore를 생성한다)

(* root로 spark-shell등을 실행하고 난후 일반 user계정으로 실행하면 이런 문제가 발생하므로 root로 spark-shell을 실행하지 말것...)

1. 실행한 사용자가 derby metastore database에 write권한이 없어서 오류가 발생하는 경우

-> cd $HOME/spark/bin

-> chmod -R a+rwx metastore_db

2. 두번째 derby는 단일 사용자가 접근이 가능해서 오류가 발생하는 경우

 -> cd $HOME/spark/bin/metastore_db

 -> rm *.lck

* hive와의 연계없이 spark-shell을 사용하는 경우라면 무시하고 사용해도된다.

* hive를 사용하는 경우라면 hive에 메타정보를 derby가 아닌 RDBMS(예, mariaDB)에 저정되도록 수정해준다.

-----------------------------오류내용----------------------------------

Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.

org.datanucleus.exceptions.NucleusDataStoreException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.

        at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:516)

        at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:298)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)

        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)

        at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)

        at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)

        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)

        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)

        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)

        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)

        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)

        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)

        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)

        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)

        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)

        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

        at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:57)

        at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:624)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)

        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)

        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)

        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)

        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)