What is Behavior Driven Development
Behavior Driven Development is a testing methodology that transpires from the TDD i.e. Test Driven Development which allows the users to work with multiple test data with minimum intervention in the software code and thereby helps to increase the reusability of the code. TDD represents a big improvement in software development and testing automations as it’s a time-saving mechanism.
Cucumber BDD
Cucumber is a tool that allows developers to write and execute human-readable acceptance tests in order to validate that a software application is working as expected. These acceptance tests, often called “features” in Cucumber, are written in a language called Gherkin and follow a specific syntax.
Gherkin syntax consists of a set of keywords that describe the behavior of the system being tested. For example, the keyword “Given” is used to set up the initial context for a test, “When” is used to describe the action being taken, and “Then” is used to describe the expected outcome. This syntax helps to create a clear and concise description of the behavior being tested, which can be understood by both technical and non-technical team members.
One of the key benefits of using Cucumber is that it allows collaboration between developers, testers, and business analysts. By using a common language (Gherkin) to describe the behavior of the system, team members from different backgrounds can work together to define and validate the acceptance criteria for a software feature.
In addition to facilitate collaboration, Cucumber also helps to ensure that the software being developed meets the needs of the end user. By writing acceptance tests in a way that is understandable to non-technical team members, Cucumber helps to ensure that the software being developed is aligned with the business goals and objectives.
Overall, Cucumber is a powerful tool for behavior driven development. It can help teams delivering high-quality software that meets the end user’s needs. By using Cucumber to define and validate acceptance criteria, teams can collaborate effectively and build software that is aligned with business goals and objectives.
Spark Scala hands-on
First, you will need to add the following dependencies to your project’s build.sbt file:
libraryDependencies ++= Seq(
"io.cucumber" % "cucumber-scala" % "6.8.1",
"io.cucumber" % "cucumber-junit" % "6.8.1" % Test
)
Next, you will need to create a feature file that defines the acceptance criteria for your Spark job. Here is an example that calculates the average salary for a group of employees:
Feature: Calculate average salary for a group of employees
Scenario: Calculate average salary for a group of employees
Given a group of employees with the following salaries:
| employee_id | salary |
| 1 | 10000 |
| 2 | 20000 |
| 3 | 30000 |
When I run the Spark job to calculate the average salary
Then the average salary should be 20000
To implement this feature, you will need to create a Scala class that defines the steps in the feature file. Here is an example of how you might implement the steps in the above feature file:
class CalculateAverageSalarySteps {
private val spark = SparkSession.builder().appName("Calculate Average Salary").getOrCreate()
import spark.implicits._
private var employeeSalaries: DataFrame = _
Given("""^a group of employees with the following salaries:$""") { (salaries: DataTable) =>
val salaryRows: List[SalaryRow] = salaries.asScala.map { row =>
val employeeId: Int = row.get("employee_id").toInt
val salary: Int = row.get("salary").toInt
SalaryRow(employeeId, salary)
}.toList
employeeSalaries = spark.createDataFrame(salaryRows)
}
When("""^I run the Spark job to calculate the average salary$""") { () =>
val averageSalary: Double = employeeSalaries.select(avg("salary")).first().getDouble(0)
}
Then("""^the average salary should be (\\d+)$""") { (expectedAverageSalary: Int) =>
assert(averageSalary == expectedAverageSalary)
}
}
Finally, you can create a Cucumber test runner class to execute the acceptance tests defined in the feature file:
@RunWith(classOf[Cucumber])
@CucumberOptions(
features = Array("src/test/resources/features"),
glue = Array("steps"),
plugin = Array("pretty", "html:target/cucumber")
)
class CalculateAverageSalaryTestRunner
To run the acceptance tests, you can simply execute the CalculateAverageSalaryTestRunner class as a JUnit test. Cucumber will execute the steps defined in the feature file and verify that the Spark job is behaving as expected.
Conclusion
In conclusion, using BDD (Behavior Driven Development) can help teams to define and validate the acceptance criteria in a way that is understandable to both technical and non-technical team members. Using BDD with Spark can also help teams to catch and fix defects early in the development process, which can save time and resources in the long run.
Overall, BDD can be a valuable tool for teams working with Apache Spark, helping them to build high-quality, reliable Spark jobs that meet the needs of the end user.
If you enjoy reading stories like these and want to support me as a writer, consider following me to not miss any new article and never forget to always enjoy learning 💡
[Omar LARAQUI - Medium
Read writing from Omar LARAQUI on Medium. Cloud Data Engineer 🚀 Double degree engineering student (ENSIMAG-ENSIAS) 🤖…medium.com](https://medium.com/@omarlaraqui "medium.com/@omarlaraqui")
[Omaroid - Overview
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…github.com](https://github.com/Omaroid "github.com/Omaroid")
https://www.linkedin.com/in/omar-laraqui
[BDD Testing & Collaboration Tools for Teams | Cucumber
Start your free trial today Bridge the gap between business and development using BDD Decrease rework with test…cucumber.io](https://cucumber.io/ "cucumber.io")
[Spark 3.3.1 ScalaDoc - org.apache.spark
Spark 3.3.1 ScalaDoc - org.apache.sparkspark.apache.org](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html "spark.apache.org/docs/latest/api/scala/org/..")