问题描述:
sparkstructedstreaming 将kafka数据写入hudi 并与hive同步
执行报错如下:
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.adapter.Spark3Adapter
解决办法:
问题显示没有发现适配器这个包?
spark也没这个包啊?
查看启动脚本,忘记添加hudi --jars了…
--jars /opt/soft/hudi/hudi-0.9.0/packaging/hudi-spark-bundle/target/hudi-spark3-bundle_2.12-0.9.0.jar --packages org.apache.spark:spark-avro_2.12:3.0.2
spark-hudi提交yarn测试完整脚本:
详细地址及包版本替换自己的即可
/opt/soft/spark3/bin/spark-submit --master yarn --deploy-mode client --driver-memory 1g --num-executors 2 --total-executor-cores 2 --executor-memory 2g --queue spark --class streaming.TestStructStreaming --jars /opt/soft/hudi/hudi-0.9.0/packaging/hudi-spark-bundle/target/hudi-spark3-bundle_2.12-0.9.0.jar --packages org.apache.spark:spark-avro_2.12:3.0.2 streaming-1.0-SNAPSHOT-jar-with-dependencies.jar