Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.
아래의 소스로 테스트할때 textFile() 부분에서 java.net.URISyntaxException가 발생하는데 이때는
아래 주석부분에 있는것처럼 sparkConf.set("spark.sql.warehouse.dir", "file:///tmp/spark-warehouse"); 를 추가하여 준다.
---------테스트 소스중 일부 --------
SparkConf sparkConf=new SparkConf().setAppName("SparkJavaTest");
sparkConf.setMaster("local[*]");
//sparkConf.set("spark.sql.warehouse.dir", "file:///tmp/spark-warehouse");
JavaSparkContext ctx=new JavaSparkContext(sparkConf);
SQLContext sqlContext = new SQLContext(ctx);
JavaRDD<String> dataCollection=ctx.textFile("file:///C:/tmp/sparktest/a.txt");
--------------------오류내용-----------------------
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:C:/dev/workspace/sdaf/sda-client/spark-warehouse
at org.apache.hadoop.fs.Path.initialize(Path.java:205)
at org.apache.hadoop.fs.Path.<init>(Path.java:171)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeQualifiedPath(SessionCatalog.scala:114)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:145)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:89)
at org.apache.spark.sql.internal.SessionState.catalog$lzycompute(SessionState.scala:95)
at org.apache.spark.sql.internal.SessionState.catalog(SessionState.scala:95)
at org.apache.spark.sql.internal.SessionState$$anon$1.<init>(SessionState.scala:112)
at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:112)
at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:111)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:542)
at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:302)
at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:315)
at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:440)
at com.gooper.icbms.sda.kafka.onem2m.SparkJavaTest.main(SparkJavaTest.java:106)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:C:/dev/workspace/sdaf/sda-client/spark-warehouse
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.<init>(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:202)
... 16 more