Fluxnet model training¶

To create fluxnet models efficiently, we created a workflow runner.

After specifying some directories, which datasets you want to include, and which labels from these datasets you'd want to use, you can run the workflow.

We'll start by importing the necessary modules and setting up Dask:

In [1]:

from pathlib import Path

from dask.distributed import Client

from excited_workflow.train_fluxnet_models import FluxnetExperiment
from excited_workflow.train_fluxnet_models import calculate_era5_derived_vars
from excited_workflow.train_fluxnet_models import collect_training_data
from excited_workflow.train_fluxnet_models import run_workflow


client = Client(n_workers=2, threads_per_worker=2)

from pathlib import Path from dask.distributed import Client from excited_workflow.train_fluxnet_models import FluxnetExperiment from excited_workflow.train_fluxnet_models import calculate_era5_derived_vars from excited_workflow.train_fluxnet_models import collect_training_data from excited_workflow.train_fluxnet_models import run_workflow client = Client(n_workers=2, threads_per_worker=2)

Next we have to define some directories:

• where is the pre-processed fluxnet data stored?
• where should the pre-processed ERA5 data be stored?
• where do you want the trained models to be written to?

Additionally, you have to define which additional (monthly) datasets are required:

In [2]:

ameriflux_file = Path("/data/volume_2/NEE_ameriflux_transcom2.nc")
preprocessed_dir = Path("/data/volume_2/preprocessed_site_data")
output_directory = Path("/data/volume_2/trained_models")

additional_datasets = [
    "biomass",
    "spei",
    "modis",
]

ameriflux_file = Path("/data/volume_2/NEE_ameriflux_transcom2.nc") preprocessed_dir = Path("/data/volume_2/preprocessed_site_data") output_directory = Path("/data/volume_2/trained_models") additional_datasets = [ "biomass", "spei", "modis", ]

If you want to know which variables will be available when you run this workflow, you can load the dataset that the workflow uses. Note that loading in all this data takes some time, especially if the ERA5 data has not been pre-processed yet.

The collect_training_data can be provided with a function that derives variables from the collected xarray Dataset. Here we use the calculate_era5_derived_vars function from excited_workflow.train_fluxnet_models.

In [3]:

ds = collect_training_data(
    ameriflux_file, preprocessed_dir, additional_datasets,
    variable_derivation=calculate_era5_derived_vars,
)
ds

ds = collect_training_data( ameriflux_file, preprocessed_dir, additional_datasets, variable_derivation=calculate_era5_derived_vars, ) ds

Valid file fluxnet-sites_era5_10m_v_component_of_wind_2004.nc already exists, skipping.                         

Out[3]:

<xarray.Dataset>
Dimensions:                         (time: 271755, site: 61)
Coordinates:
  * time                            (time) datetime64[ns] 1991-01-01T06:00:00...
  * site                            (site) object 'US-Rws' 'US-ARM' ... 'US-KLS'
Data variables: (12/29)
    GPP_NT_VUT_REF                  (site, time) float64 nan nan nan ... nan nan
    GPP_DT_VUT_REF                  (site, time) float64 nan nan nan ... nan nan
    NEE_VUT_REF                     (site, time) float64 nan nan nan ... nan nan
    latitude                        (site) float64 43.17 36.61 ... 45.56 38.77
    longitude                       (site) float64 -116.7 -97.49 ... -97.57
    resp                            (site, time) float64 nan nan nan ... nan nan
    ...                              ...
    day_of_year                     (time) int64 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1
    hour                            (time) int64 6 7 8 9 10 11 ... 3 4 5 6 7 8
    t2m_1w_rolling                  (site, time) float32 dask.array<chunksize=(61, 43902), meta=np.ndarray>
    mean_air_temperature            (site) float32 dask.array<chunksize=(61,), meta=np.ndarray>
    mean_dewpoint_depression        (site) float32 dask.array<chunksize=(61,), meta=np.ndarray>
    dewpoint_depression_1w_rolling  (site, time) float32 dask.array<chunksize=(61, 43902), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • time: 271755
  • site: 61
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    1991-01-01T06:00:00 ... 2022-01-...

    array(['1991-01-01T06:00:00.000000000', '1991-01-01T07:00:00.000000000',
           '1991-01-01T08:00:00.000000000', ..., '2022-01-01T06:00:00.000000000',
           '2022-01-01T07:00:00.000000000', '2022-01-01T08:00:00.000000000'],
          dtype='datetime64[ns]')

  • site
    (site)
    object
    'US-Rws' 'US-ARM' ... 'US-KLS'

    array(['US-Rws', 'US-ARM', 'US-CS4', 'US-Tw3', 'US-Hn3', 'US-Ho2', 'US-KS3',
           'US-Rwe', 'US-Me2', 'US-Mpj', 'US-MOz', 'US-HWB', 'US-SRC', 'CA-DB2',
           'US-Tw2', 'US-Bi2', 'US-CF2', 'US-Wjs', 'US-OWC', 'US-Ha1', 'US-KFS',
           'US-CF4', 'US-xBR', 'CA-TPD', 'US-CS1', 'CA-Cbo', 'US-Sne', 'MX-Tes',
           'US-Ro5', 'US-Bi1', 'US-ONA', 'US-Ro1', 'US-CF1', 'US-Ro4', 'CA-DBB',
           'US-Tw1', 'US-Rwf', 'US-Rms', 'US-NC1', 'US-CS3', 'US-Snf', 'US-GLE',
           'US-A32', 'CA-TP3', 'US-UM3', 'CA-ER1', 'US-Jo2', 'US-Hn2', 'US-CF3',
           'US-Jo1', 'US-Myb', 'US-HB1', 'US-MMS', 'US-NR1', 'US-Ro6', 'US-CS2',
           'US-Ne1', 'US-Tw5', 'US-ORv', 'US-UMd', 'US-KLS'], dtype=object)

• Data variables: (29)
  • GPP_NT_VUT_REF
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    units :

    umolCO2 m-2 s-1

    long_name :

    gross_primary_production

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • GPP_DT_VUT_REF
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • NEE_VUT_REF
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    units :

    umolCO2 m-2 s-1

    long_name :

    net_ecosystem_exchange

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • latitude
    (site)
    float64
    43.17 36.61 44.16 ... 45.56 38.77

    array([43.1675, 36.6058, 44.1597, 38.1151, 46.6878, 45.2091, 28.7084,
           43.0653, 44.4523, 34.4384, 38.7441, 40.8608, 31.9083, 49.119 ,
           38.1047, 38.1091, 46.784 , 34.4255, 41.3795, 42.5378, 39.0561,
           46.7518, 44.0639, 42.6353, 44.1031, 44.3167, 38.0369, 27.84  ,
           44.691 , 38.0992, 27.3836, 44.7143, 46.7815, 44.6781, 49.1293,
           38.1074, 43.1207, 43.0645, 35.8118, 44.1394, 38.0402, 41.3665,
           36.8193, 42.7068, 45.5686, 43.6405, 32.5849, 46.6889, 46.7551,
           32.582 , 38.0498, 33.3455, 39.3232, 40.0329, 44.6946, 44.1467,
           41.1651, 38.1072, 40.0201, 45.5625, 38.7745])

  • longitude
    (site)
    float64
    -116.7 -97.49 ... -84.7 -97.57

    array([-116.7132,  -97.4888,  -89.5475, -121.6467, -119.4614,  -68.747 ,
            -80.7427, -116.7591, -121.5574, -106.2377,  -92.2   ,  -77.8488,
           -110.8395, -122.9951, -121.6433, -121.5351, -117.0908, -105.8615,
            -82.5125,  -72.1715,  -95.1907, -117.1285,  -71.2873,  -80.5577,
            -89.5379,  -79.9333, -121.7547, -109.3   ,  -93.0576, -121.4993,
            -81.9509,  -93.0898, -117.0821,  -93.0723, -122.9849, -121.6469,
           -116.7231, -116.7486,  -76.7119,  -89.5727, -121.7272, -106.2399,
            -97.8198,  -80.3483,  -84.6707,  -80.4123, -106.6032, -119.4641,
           -117.1261, -106.635 , -121.7651,  -79.1957,  -86.4131, -105.5464,
            -93.0578,  -89.5002,  -96.4766, -121.6426,  -83.0183,  -84.6975,
            -97.5684])

  • resp
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    units :

    umolCO2 m-2 s-1

    long_name :

    ecosystem_respiration

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • d2m
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    K

    long_name :

    2 metre dewpoint temperature

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • mslhf
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    W m**-2

    long_name :

    Mean surface latent heat flux

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • msshf
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    W m**-2

    long_name :

    Mean surface sensible heat flux

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • sp
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    Pa

    long_name :

    Surface pressure

    standard_name :

    surface_air_pressure

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • ssr
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    J m**-2

    long_name :

    Surface net short-wave (solar) radiation

    standard_name :

    surface_net_downward_shortwave_flux

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • ssrd
    (site, time)
    float32
    dask.array<chunksize=(61, 131490), meta=np.ndarray>

    units :

    J m**-2

    long_name :

    Surface short-wave (solar) radiation downwards

    standard_name :

    surface_downwelling_shortwave_flux_in_air

    Array Chunk Bytes 63.24 MiB 30.60 MiB Shape (61, 271755) (61, 131490) Dask graph 16 chunks in 47 graph layers Data type float32 numpy.ndarray

  • str
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    J m**-2

    long_name :

    Surface net long-wave (thermal) radiation

    standard_name :

    surface_net_upward_longwave_flux

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • strd
    (site, time)
    float32
    dask.array<chunksize=(61, 131490), meta=np.ndarray>

    units :

    J m**-2

    long_name :

    Surface long-wave (thermal) radiation downwards

    Array Chunk Bytes 63.24 MiB 30.60 MiB Shape (61, 271755) (61, 131490) Dask graph 16 chunks in 47 graph layers Data type float32 numpy.ndarray

  • t2m
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    K

    long_name :

    2 metre temperature

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • tp
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    m

    long_name :

    Total precipitation

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • u10
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    m s**-1

    long_name :

    10 metre U wind component

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • v10
    (site, time)
    float32
    dask.array<chunksize=(61, 43818), meta=np.ndarray>

    units :

    m s**-1

    long_name :

    10 metre V wind component

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43818) Dask graph 26 chunks in 61 graph layers Data type float32 numpy.ndarray

  • biomass
    (site, time)
    float64
    nan nan nan ... 1.594 1.594 1.594

    long_name :

    annual carbon density map of global live woody vegetation

    units :

    Mg ha-1

    array([[        nan,         nan,         nan, ...,  9.34228785,
             9.34228785,  9.34228785],
           [        nan,         nan,         nan, ...,  3.10616469,
             3.10616469,  3.10616469],
           [        nan,         nan,         nan, ..., 31.48978667,
            31.48978667, 31.48978667],
           ...,
           [        nan,         nan,         nan, ..., 21.44913098,
            21.44913098, 21.44913098],
           [        nan,         nan,         nan, ..., 50.35534514,
            50.35534514, 50.35534514],
           [        nan,         nan,         nan, ...,  1.59376207,
             1.59376207,  1.59376207]])

  • spei
    (site, time)
    float64
    -1.502 -1.502 -1.502 ... nan nan

    units :

    z-values

    long_name :

    Standardized Precipitation-Evapotranspiration Index

    array([[-1.50236154, -1.50236154, -1.50236154, ...,         nan,
                    nan,         nan],
           [-0.76313591, -0.76313591, -0.76313591, ...,         nan,
                    nan,         nan],
           [ 0.30765419,  0.30765419,  0.30765419, ...,         nan,
                    nan,         nan],
           ...,
           [ 1.64370587,  1.64370587,  1.64370587, ...,         nan,
                    nan,         nan],
           [ 0.01676631,  0.01676631,  0.01676631, ...,         nan,
                    nan,         nan],
           [-0.95879525, -0.95879525, -0.95879525, ...,         nan,
                    nan,         nan]])

  • NDVI
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    long_name :

    normalized difference vegetation index

    units :

    -

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • NIRv
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    long_name :

    near-infrared reflectance of vegetation

    units :

    -

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • ssr_6hr
    (site, time)
    float32
    dask.array<chunksize=(61, 43821), meta=np.ndarray>

    units :

    J m**-2

    long_name :

    Surface net short-wave (solar) radiation

    standard_name :

    surface_net_downward_shortwave_flux

    Array Chunk Bytes 63.24 MiB 10.20 MiB Shape (61, 271755) (61, 43821) Dask graph 27 chunks in 88 graph layers Data type float32 numpy.ndarray

  • day_of_year
    (time)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    units :

    day

    long_name :

    day of year

    array([1, 1, 1, ..., 1, 1, 1])

  • hour
    (time)
    int64
    6 7 8 9 10 11 12 ... 2 3 4 5 6 7 8

    array([6, 7, 8, ..., 6, 7, 8])

  • t2m_1w_rolling
    (site, time)
    float32
    dask.array<chunksize=(61, 43902), meta=np.ndarray>

    units :

    K

    long_name :

    2 metre temperature

    Array Chunk Bytes 63.24 MiB 10.22 MiB Shape (61, 271755) (61, 43902) Dask graph 27 chunks in 88 graph layers Data type float32 numpy.ndarray

  • mean_air_temperature
    (site)
    float32
    dask.array<chunksize=(61,), meta=np.ndarray>

    units :

    K

    long_name :

    2 metre temperature

    Array Chunk Bytes 244 B 244 B Shape (61,) (61,) Dask graph 1 chunks in 65 graph layers Data type float32 numpy.ndarray

  • mean_dewpoint_depression
    (site)
    float32
    dask.array<chunksize=(61,), meta=np.ndarray>

    units :

    K

    long_name :

    Dewpoint depression

    Array Chunk Bytes 244 B 244 B Shape (61,) (61,) Dask graph 1 chunks in 125 graph layers Data type float32 numpy.ndarray

  • dewpoint_depression_1w_rolling
    (site, time)
    float32
    dask.array<chunksize=(61, 43902), meta=np.ndarray>

    units :

    K

    long_name :

    Dewpoint depression

    Array Chunk Bytes 63.24 MiB 10.22 MiB Shape (61, 271755) (61, 43902) Dask graph 27 chunks in 165 graph layers Data type float32 numpy.ndarray

• Indexes: (2)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1991-01-01 06:00:00', '1991-01-01 07:00:00',
                   '1991-01-01 08:00:00', '1991-01-01 09:00:00',
                   '1991-01-01 10:00:00', '1991-01-01 11:00:00',
                   '1991-01-01 12:00:00', '1991-01-01 13:00:00',
                   '1991-01-01 14:00:00', '1991-01-01 15:00:00',
                   ...
                   '2021-12-31 23:00:00', '2022-01-01 00:00:00',
                   '2022-01-01 01:00:00', '2022-01-01 02:00:00',
                   '2022-01-01 03:00:00', '2022-01-01 04:00:00',
                   '2022-01-01 05:00:00', '2022-01-01 06:00:00',
                   '2022-01-01 07:00:00', '2022-01-01 08:00:00'],
                  dtype='datetime64[ns]', name='time', length=271755, freq='H'))

  • site
    PandasIndex

    PandasIndex(Index(['US-Rws', 'US-ARM', 'US-CS4', 'US-Tw3', 'US-Hn3', 'US-Ho2', 'US-KS3',
           'US-Rwe', 'US-Me2', 'US-Mpj', 'US-MOz', 'US-HWB', 'US-SRC', 'CA-DB2',
           'US-Tw2', 'US-Bi2', 'US-CF2', 'US-Wjs', 'US-OWC', 'US-Ha1', 'US-KFS',
           'US-CF4', 'US-xBR', 'CA-TPD', 'US-CS1', 'CA-Cbo', 'US-Sne', 'MX-Tes',
           'US-Ro5', 'US-Bi1', 'US-ONA', 'US-Ro1', 'US-CF1', 'US-Ro4', 'CA-DBB',
           'US-Tw1', 'US-Rwf', 'US-Rms', 'US-NC1', 'US-CS3', 'US-Snf', 'US-GLE',
           'US-A32', 'CA-TP3', 'US-UM3', 'CA-ER1', 'US-Jo2', 'US-Hn2', 'US-CF3',
           'US-Jo1', 'US-Myb', 'US-HB1', 'US-MMS', 'US-NR1', 'US-Ro6', 'US-CS2',
           'US-Ne1', 'US-Tw5', 'US-ORv', 'US-UMd', 'US-KLS'],
          dtype='object', name='site'))

• Attributes: (0)

Additionally, if you want to get the pandas DataFrame that is used for training the model, run the following command; ds.to_dataframe().dropna().reset_index()

Defining the experiments¶

Now you can define your model training parameters. You will need to define:

• The group_key: this is the name of the variable used for splitting up data in cross-validation (i.e. the site names).
• The predictor variables (X_keys).
• The target variable (y_key)
• the cross validation method (see the scikit-learn documentation)
• The name of the ML model you want to use. For the available ones, see the documentation on pycaret

In [4]:

# Define keys for models
group_key = "site"  # for fold groups
X_keys_resp = [
    "d2m", "t2m", "ssr", # era5
    "biomass", "spei", "NDVI", "NIRv", # other datasets
    "day_of_year", "t2m_1w_rolling", "mean_air_temperature",
    "mean_dewpoint_depression", "dewpoint_depression_1w_rolling"
]
y_key_resp = "resp"

X_keys_gpp = [
    "d2m", "mslhf", "msshf", "ssr", "ssr_6hr", "str", "t2m", # era5
    "biomass", "spei", "NDVI", "NIRv", # other datasets
]
y_key_gpp = "GPP_NT_VUT_REF"

from sklearn.model_selection import GroupShuffleSplit
cv_method = GroupShuffleSplit(n_splits=10, test_size=0.4)

# Define keys for models group_key = "site" # for fold groups X_keys_resp = [ "d2m", "t2m", "ssr", # era5 "biomass", "spei", "NDVI", "NIRv", # other datasets "day_of_year", "t2m_1w_rolling", "mean_air_temperature", "mean_dewpoint_depression", "dewpoint_depression_1w_rolling" ] y_key_resp = "resp" X_keys_gpp = [ "d2m", "mslhf", "msshf", "ssr", "ssr_6hr", "str", "t2m", # era5 "biomass", "spei", "NDVI", "NIRv", # other datasets ] y_key_gpp = "GPP_NT_VUT_REF" from sklearn.model_selection import GroupShuffleSplit cv_method = GroupShuffleSplit(n_splits=10, test_size=0.4)

All this information has to be provided to the FluxnetExperiment 'dataclass'.

We will train three models here, two for the respiration (using different ML models) and one for GPP.

The name of the experiment is used to create the model's output directory, you're free to define this.

In [5]:

models = [
    FluxnetExperiment(
        name="respiration",
        X_keys=X_keys_resp,
        y_key=y_key_resp,
        ml_model_name="ridge",
        cv_method=cv_method,
        cv_group_key=group_key,
        output_dir=output_directory
    ),
    FluxnetExperiment(
        name="respiration",
        X_keys=X_keys_resp,
        y_key=y_key_resp,
        ml_model_name="lightgbm",
        cv_method=cv_method,
        cv_group_key=group_key,
        output_dir=output_directory
    ),
    FluxnetExperiment(
        name="gpp",
        X_keys=X_keys_gpp,
        y_key=y_key_gpp,
        ml_model_name="lightgbm",
        cv_method=cv_method,
        cv_group_key=group_key,
        output_dir=output_directory
    ),
]

models = [ FluxnetExperiment( name="respiration", X_keys=X_keys_resp, y_key=y_key_resp, ml_model_name="ridge", cv_method=cv_method, cv_group_key=group_key, output_dir=output_directory ), FluxnetExperiment( name="respiration", X_keys=X_keys_resp, y_key=y_key_resp, ml_model_name="lightgbm", cv_method=cv_method, cv_group_key=group_key, output_dir=output_directory ), FluxnetExperiment( name="gpp", X_keys=X_keys_gpp, y_key=y_key_gpp, ml_model_name="lightgbm", cv_method=cv_method, cv_group_key=group_key, output_dir=output_directory ), ]

Executing the workflow¶

Now you can run the workflow. For a deeper look into the specific steps of the workflow, see the file src/excited_workflow/train_fluxnet_models.py.

The workflow will create new folders inside the specified output directory (output_directory defined in cell 2).

Each folder contains:

• a model description file (in markdown)
• validation plots
• a JSON file with the used variables (and attributes)
• the model ONNX file

This should make it easier to asses the model training and use the model to produce a dataset.

In [6]:

run_workflow(
    fluxnet_file=ameriflux_file,
    preprocessing_dir=preprocessed_dir,
    additional_datasets=additional_datasets,
    models=models,
    variable_derivation=calculate_era5_derived_vars,
)

run_workflow( fluxnet_file=ameriflux_file, preprocessing_dir=preprocessed_dir, additional_datasets=additional_datasets, models=models, variable_derivation=calculate_era5_derived_vars, )

Valid file fluxnet-sites_era5_10m_v_component_of_wind_2004.nc already exists, skipping.                         

	Model	MAE	MSE	RMSE	R2	RMSLE	MAPE	TT (Sec)
ridge	Ridge Regression	1.53713	6.32171	2.47449	0.36426	0.50615	19573.32939	0.73600

	Model	MAE	MSE	RMSE	R2	RMSLE	MAPE	TT (Sec)
lightgbm	Light Gradient Boosting Machine	1.39383	5.39083	2.28724	0.34206	0.46973	16018.56091	3.79500

The maximum opset needed by this model is only 8.

	Model	MAE	MSE	RMSE	R2	RMSLE	MAPE	TT (Sec)
lightgbm	Light Gradient Boosting Machine	2.40524	23.23889	4.81172	0.56179	0.62698	7.12475	3.14500

The maximum opset needed by this model is only 8.

In [ ]: