Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다.
문의사항은 gooper@gooper.com로 메일을
보내주세요.
1. mysql을 설치/설정한다.(hive-site.xml)
: hive.metastore.lcoal은 설정하지 않아도됨
-------------------------------------------------hive-site.xml의 일부--------------------------------------------------------
<property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.8.5:9083</value>
<description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property><property>
<name>hive.metastore.local</name>
<value>false</value>
</property><property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/metastore_db</value>
<description>JDBC connect string for a JDBC metastore</description>
</property><property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hadoop/warehouse</value>
</property><property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>2. Hive metastore interface를 기동해준다.bin/hive --service metastore &* hive metastore interface기동하지 않고 oozie에서 hive job을 돌리면...아래와 같은 오류가 발생하며 job이 kill된다.=================================================================
>>> Invoking Hive command line now >>>
Hadoop Job IDs executed by Hive:
Intercepting System.exit(40000)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher ends
-- End of task: attempt_201405271904_0020_m_000000_0 --
stderr logs
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
Logging initialized using configuration in jar:file:/tmp/mapr-hadoop/mapred/local/taskTracker/distcache/5455234734278781531_309545739_993904106/maprfs/mapr/my.cluster.com/lib/hive/hive-common-0.12-mapr-1401-140130.jar!/hive-log4j.properties
FAILED: SemanticException Unable to fetch table ht_dm_101010
Log file: /tmp/mapr-hadoop/mapred/local/taskTracker/mapr/jobcache/job_201405271904_0020/attempt_201405271904_0020_m_000000_0/work/hive-oozie-job_201405271904_0020.log not present. Therefore no Hadoop jobids found
Intercepting System.exit(40000)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]
-- End of task: attempt_201405271904_0020_m_000000_0 --