Packaging scala class on databricks (error: not found: value dbutils)

Question

尝试用 class

制作一个包裹

package x.y.Log


import scala.collection.mutable.ListBuffer
import org.apache.spark.sql.{DataFrame}
import org.apache.spark.sql.functions.{lit, explode, collect_list, struct}
import org.apache.spark.sql.types.{StructField, StructType}
import java.util.Calendar
import java.text.SimpleDateFormat
import org.apache.spark.sql.functions._
import spark.implicits._

class Log{
...
}

在同一个笔记本上一切正常，但是当我尝试创建可以在其他笔记本上使用的包时，我遇到了错误：

<notebook>:11: error: not found: object spark
import spark.implicits._
       ^
<notebook>:21: error: not found: value dbutils
  val notebookPath = dbutils.notebook.getContext().notebookPath.get
                     ^
<notebook>:22: error: not found: value dbutils
  val userName = dbutils.notebook.getContext.tags("user")
                 ^
<notebook>:23: error: not found: value dbutils
  val userId = dbutils.notebook.getContext.tags("userId")
               ^
<notebook>:41: error: not found: value spark
    var rawMeta =  spark.read.format("json").option("multiLine", true).load("/FileStore/tables/xxx.json")
                   ^
<notebook>:42: error: value $ is not a member of StringContext
    .filter($"Name".isin(readSources))

有人知道如何用这些库打包这个 class 吗？

Answer 1

假设您是运行 Spark 2.x，语句 import spark.implicits._ 仅在范围内有 SparkSession 对象时才有效。对象 Implicits 在 SparkSession 对象中定义。该对象扩展了以前版本的 spark Link to SparkSession code on Github 的 SQLImplicits。您可以检查 link 以验证

package x.y.Log


import scala.collection.mutable.ListBuffer
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.{lit, explode, collect_list, struct}
import org.apache.spark.sql.types.{StructField, StructType}
import java.util.Calendar
import java.text.SimpleDateFormat
import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession

class Log{

  val spark: SparkSession = SparkSession.builder.enableHiveSupport().getOrCreate()

  import spark.implicits._

  ...[rest of the code below]
}

Packaging scala class on databricks (error: not found: value dbutils)

Packaging scala class on databricks (error: not found: value dbutils)

scala

apache-spark

databricks