Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.
1. oozie를 compile하면 oozie-sharelib-4.1.0.tar.gz가 생성되는데 이 파일을 압축해제 하여 HDFS상에 등록해서 oozie job을 실행할때
공통라이브러리로 사용하게 한다.
2. 압축풀기
tar xvfz oozie-sharelib-4.1.0.tar.gz
3. sharelib HDFS에 등록
hadoop fs -put share /user/hadoop
4. oozie job만들때 job.properties에 아래와 같이 true로 지정해준다.
두번째는 별도로 라이브러리를 지정해줄때 사용한다.
oozie.use.system.libpath=true
oozie.libpath=/user/hadoop/share/lib/hive
5. 위와 같이하면 "File /user/hadoop/share/lib does not exist"와 같은 오류가 발생하는데 이때는 oozie-site.mxl을 수정해준다.
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<!-- value>*=hadoop-conf</value -->
<value>*=/hadoop/hadoop/etc/hadoop/</value>
<description>
Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
the relevant Hadoop *-site.xml files. If the path is relative is looked within
the Oozie configuration directory; though the path can be absolute (i.e. to point
to Hadoop client conf/ directories in the local filesystem.
</description>
</property>
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<!-- value>/user/${user.name}/share/lib</value -->
<value>hdfs:///user/${user.name}/share/lib</value>
<description>
System library path to use for workflow applications.
This path is added to workflow application if their job properties sets
the property 'oozie.use.system.libpath' to true.
</description>
</property>
6. oozie 재기동
oozied.sh restart