Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.
spark "Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources"오류 발생시 조치사항
spark-shell등이나 spark어플리케이션을 실행할때 "Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources"라는 warnning이 발생되면서 어플리케이션이 수행되지 않고 대기하고 http://sda1:8123하였을때 "Cores in use: 40 Total,40 Used"으로 core를 모두 사용하고 있고, "Running Application"s의 status가 WAITING인 상황이 발생한다.
이때 아래의 설정값을 conf/spark-env.conf에 지정하여 어플리케이션에서 사용할 수 있는 core와 memory를 제한하도록 설정한다.
export SPARK_WORKER_INSTANCES=4(일반적으로 1로 지정할것)
export SPARK_WORKER_MEMORY=5000m
export SPARK_WORKER_CORES=10
export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=3"
==> 위에 옵션은 start-all.sh할때 각서버에 worker가 4개씩 기동하며 각 worker당 최대사용가능한 core수 10개, 메모리는 5000m씩 사용하게 된다. 특히 SPARK_MASTER_OPTS을 지정하면 어플리케이션에서 사용가능한(예시에서는 3개) core의 수를 제한하게 되어 다른 어플리케이션이 resource를 사용할 수 있는 여유가 생기게 된다. 하지만 stop-all.sh로 중지시키면 서버당 4개씩 기동된 Worker를 모두 중지시켜 주지못하는 문제점이 있다.
-----------------------------WARN내용----------------------------------
16/05/24 18:29:27 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:29:42 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:29:57 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:30:12 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:30:27 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:30:42 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:30:57 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:31:12 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:31:27 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/05/24 18:31:42 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources