必须包括log4j的,但它造成的Apache星火外壳的错误。如何避免错误? [英] Must include log4J, but it is causing errors in Apache Spark shell. How to avoid errors?
问题描述
由于一个,我必须包括成火花code坛子里的复杂性,我想问一下搞清楚的方式来解决这个问题,无需拆卸进口log4j的帮助。
Due to a complexity of the jars that I must include into a Spark code, I would like to ask for a help figuring out the way to solve this issue without removing the log4j import.
简单code是如下:
:cp symjar/log4j-1.2.17.jar
import org.apache.spark.rdd._
val hadoopConf=sc.hadoopConfiguration;
hadoopConf.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3n.awsAccessKeyId","AKEY")
hadoopConf.set("fs.s3n.awsSecretAccessKey","SKEY")
val numOfProcessors = 2
val filePath = "s3n://SOMEFILE.csv"
var rdd = sc.textFile(filePath, numOfProcessors)
def doStuff(rdd: RDD[String]): RDD[String] = {rdd}
doStuff(rdd)
首先,我得到这个错误:
First, I am getting this error:
error: error while loading StorageLevel, class file '/root/spark/lib/spark-assembly-1.3.0-hadoop1.0.4.jar(org/apache/spark/storage/StorageLevel.class)' has location not matching its contents: contains class StorageLevel
error: error while loading Partitioner, class file '/root/spark/lib/spark-assembly-1.3.0-hadoop1.0.4.jar(org/apache/spark/Partitioner.class)' has location not matching its contents: contains class Partitioner
error: error while loading BoundedDouble, class file '/root/spark/lib/spark-assembly-1.3.0-hadoop1.0.4.jar(org/apache/spark/partial/BoundedDouble.class)' has location not matching its contents: contains class BoundedDouble
error: error while loading CompressionCodec, class file '/root/spark/lib/spark-assembly-1.3.0-hadoop1.0.4.jar(org/apache/hadoop/io/compress/CompressionCodec.class)' has location not matching its contents: contains class CompressionCodec
然后,我再次运行此线,误差自败:
Then, I run this line again, and the error dissapears:
var rdd = sc.textFile(filePath, numOfProcessors)
不过,code的最终结果是:
However, the end-result of the code is:
error: type mismatch;
found : org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
required: org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
doStuff(rdd)
^
我怎样才能避免进口删除log4j的,并没有得到所提到的错误? (这是项重要的,因为我有大量使用Log4j和罐子在与火花壳牌冲突)。
How can I avoid removing the log4j from the import and not get the mentioned errors ? (this is imporant, since the jars that I have use log4j heavily and are in conflict with Spark-Shell).
推荐答案
答案是不是只使用:cp命令,但也增加了包括一切都在... /火花/ conf目录/ spark-env.sh在出口的 SPARK_SUBMIT_CLASSPATH =... /的/路径/要/ a.jar文件
The answer is not to use just the :cp command but to also to add the include everything in .../spark/conf/spark-env.sh under the export SPARK_SUBMIT_CLASSPATH=".../the/path/to/a.jar"
这篇关于必须包括log4j的,但它造成的Apache星火外壳的错误。如何避免错误?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!