environment_variable.py
EnvironmentVariable
dataclass
Bases: IO[str]
IO used to load and save environment variables. Use: - as input, to parameterize the node logic - as output, to set an environment variable based on node logic
Gets and sets os.environ
on load and save. See the Python docs for more
information.
Example in a node:
main.py
from ordeq import run, node
from ordeq_spark import SparkHiveTable
import pyspark.sql.functions as F
from pyspark.sql import DataFrame
@node(
inputs=[
SparkHiveTable(table="my.table"),
EnvironmentVariable("KEY", default="DEFAULT"),
],
outputs=SparkHiveTable(table="my.output"),
)
def transform(df: DataFrame, value: str) -> DataFrame:
return df.where(F.col("col") == value)
if __name__ == "__main__":
run(transform)
When you run transform
through the CLI as follows:
export KEY=MyValue
python main.py transform
MyValue
will be used as value
in transform
.