Preprocessing Ameriflux data¶

In this notebook the Ameriflux data is prepared for further analysis.

The resulting data is an xarray Dataset, containing the required data for all sites, with the time set to UTC.

To be able to run this notebook you need:

• A folder containing Ameriflux site .zip files (e.g., AMF_US-xBR_FLUXNET_FULLSET_2017-2021_3-5.zip)
• The metadata file AMF_AA-Flx_FLUXNET-BIF_CCBY4_20221210.xlsx
• And the netCDF file describing the transcom regions.

In [1]:

from pathlib import Path
import matplotlib.pyplot as plt
from src import process_fluxnet

from pathlib import Path import matplotlib.pyplot as plt from src import process_fluxnet

Read and preprocess all sites:

In [2]:

ds = process_fluxnet.preprocess_ameriflux_sites(
    zip_folder=Path("/home/bart/Data/EXCITED/fluxnet/zips"),
    metadata_file=Path("/home/bart/Data/EXCITED/fluxnet/AMF_AA-Flx_FLUXNET-BIF_CCBY4_20221210.xlsx"),
    transcom_regions_file=Path("/home/bart/Data/EXCITED/regions.nc"),
)

ds = process_fluxnet.preprocess_ameriflux_sites( zip_folder=Path("/home/bart/Data/EXCITED/fluxnet/zips"), metadata_file=Path("/home/bart/Data/EXCITED/fluxnet/AMF_AA-Flx_FLUXNET-BIF_CCBY4_20221210.xlsx"), transcom_regions_file=Path("/home/bart/Data/EXCITED/regions.nc"), )

Found 81 Ameriflux sites.
Failed to read property. site='US-Me1', prop='LOCATION_LAT'
Failed to read property. site='US-Me1', prop='LOCATION_LONG'
Failed to read property. site='US-Me3', prop='LOCATION_LAT'
Failed to read property. site='US-Me3', prop='LOCATION_LONG'
Of which 61 are located within transcom region 2.
Starting to load the .csv files...
.csv files loaded. Merging them and creating the dataset.

The resulting xr.Dataset contains the data for all Ameriflux sites:

In [3]:

ds

ds

Out[3]:

<xarray.Dataset>
Dimensions:         (time: 271755, site: 61)
Coordinates:
  * time            (time) datetime64[ns] 1991-01-01T06:00:00 ... 2022-01-01T...
  * site            (site) <U6 'US-Rws' 'US-ARM' 'US-CS4' ... 'US-UMd' 'US-KLS'
Data variables:
    GPP_NT_VUT_REF  (site, time) float64 nan nan nan nan nan ... nan nan nan nan
    GPP_DT_VUT_REF  (site, time) float64 nan nan nan nan nan ... nan nan nan nan
    NEE_VUT_REF     (site, time) float64 nan nan nan nan nan ... nan nan nan nan
    latitude        (site) float64 43.17 36.61 44.16 38.12 ... 40.02 45.56 38.77
    longitude       (site) float64 -116.7 -97.49 -89.55 ... -83.02 -84.7 -97.57

xarray.Dataset

• Dimensions:
  • time: 271755
  • site: 61
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    1991-01-01T06:00:00 ... 2022-01-...

    array(['1991-01-01T06:00:00.000000000', '1991-01-01T07:00:00.000000000',
           '1991-01-01T08:00:00.000000000', ..., '2022-01-01T06:00:00.000000000',
           '2022-01-01T07:00:00.000000000', '2022-01-01T08:00:00.000000000'],
          dtype='datetime64[ns]')

  • site
    (site)
    <U6
    'US-Rws' 'US-ARM' ... 'US-KLS'

    array(['US-Rws', 'US-ARM', 'US-CS4', 'US-Tw3', 'US-Hn3', 'US-Ho2', 'US-KS3',
           'US-Rwe', 'US-Me2', 'US-Mpj', 'US-MOz', 'US-HWB', 'US-SRC', 'CA-DB2',
           'US-Tw2', 'US-Bi2', 'US-CF2', 'US-Wjs', 'US-OWC', 'US-Ha1', 'US-KFS',
           'US-CF4', 'US-xBR', 'CA-TPD', 'US-CS1', 'CA-Cbo', 'US-Sne', 'MX-Tes',
           'US-Ro5', 'US-Bi1', 'US-ONA', 'US-Ro1', 'US-CF1', 'US-Ro4', 'CA-DBB',
           'US-Tw1', 'US-Rwf', 'US-Rms', 'US-NC1', 'US-CS3', 'US-Snf', 'US-GLE',
           'US-A32', 'CA-TP3', 'US-UM3', 'CA-ER1', 'US-Jo2', 'US-Hn2', 'US-CF3',
           'US-Jo1', 'US-Myb', 'US-HB1', 'US-MMS', 'US-NR1', 'US-Ro6', 'US-CS2',
           'US-Ne1', 'US-Tw5', 'US-ORv', 'US-UMd', 'US-KLS'], dtype='<U6')

• Data variables: (5)
  • GPP_NT_VUT_REF
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • GPP_DT_VUT_REF
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • NEE_VUT_REF
    (site, time)
    float64
    nan nan nan nan ... nan nan nan nan

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]])

  • latitude
    (site)
    float64
    43.17 36.61 44.16 ... 45.56 38.77

    array([43.1675, 36.6058, 44.1597, 38.1151, 46.6878, 45.2091, 28.7084,
           43.0653, 44.4523, 34.4384, 38.7441, 40.8608, 31.9083, 49.119 ,
           38.1047, 38.1091, 46.784 , 34.4255, 41.3795, 42.5378, 39.0561,
           46.7518, 44.0639, 42.6353, 44.1031, 44.3167, 38.0369, 27.84  ,
           44.691 , 38.0992, 27.3836, 44.7143, 46.7815, 44.6781, 49.1293,
           38.1074, 43.1207, 43.0645, 35.8118, 44.1394, 38.0402, 41.3665,
           36.8193, 42.7068, 45.5686, 43.6405, 32.5849, 46.6889, 46.7551,
           32.582 , 38.0498, 33.3455, 39.3232, 40.0329, 44.6946, 44.1467,
           41.1651, 38.1072, 40.0201, 45.5625, 38.7745])

  • longitude
    (site)
    float64
    -116.7 -97.49 ... -84.7 -97.57

    array([-116.7132,  -97.4888,  -89.5475, -121.6467, -119.4614,  -68.747 ,
            -80.7427, -116.7591, -121.5574, -106.2377,  -92.2   ,  -77.8488,
           -110.8395, -122.9951, -121.6433, -121.5351, -117.0908, -105.8615,
            -82.5125,  -72.1715,  -95.1907, -117.1285,  -71.2873,  -80.5577,
            -89.5379,  -79.9333, -121.7547, -109.3   ,  -93.0576, -121.4993,
            -81.9509,  -93.0898, -117.0821,  -93.0723, -122.9849, -121.6469,
           -116.7231, -116.7486,  -76.7119,  -89.5727, -121.7272, -106.2399,
            -97.8198,  -80.3483,  -84.6707,  -80.4123, -106.6032, -119.4641,
           -117.1261, -106.635 , -121.7651,  -79.1957,  -86.4131, -105.5464,
            -93.0578,  -89.5002,  -96.4766, -121.6426,  -83.0183,  -84.6975,
            -97.5684])

• Indexes: (2)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1991-01-01 06:00:00', '1991-01-01 07:00:00',
                   '1991-01-01 08:00:00', '1991-01-01 09:00:00',
                   '1991-01-01 10:00:00', '1991-01-01 11:00:00',
                   '1991-01-01 12:00:00', '1991-01-01 13:00:00',
                   '1991-01-01 14:00:00', '1991-01-01 15:00:00',
                   ...
                   '2021-12-31 23:00:00', '2022-01-01 00:00:00',
                   '2022-01-01 01:00:00', '2022-01-01 02:00:00',
                   '2022-01-01 03:00:00', '2022-01-01 04:00:00',
                   '2022-01-01 05:00:00', '2022-01-01 06:00:00',
                   '2022-01-01 07:00:00', '2022-01-01 08:00:00'],
                  dtype='datetime64[ns]', name='time', length=271755, freq='H'))

  • site
    PandasIndex

    PandasIndex(Index(['US-Rws', 'US-ARM', 'US-CS4', 'US-Tw3', 'US-Hn3', 'US-Ho2', 'US-KS3',
           'US-Rwe', 'US-Me2', 'US-Mpj', 'US-MOz', 'US-HWB', 'US-SRC', 'CA-DB2',
           'US-Tw2', 'US-Bi2', 'US-CF2', 'US-Wjs', 'US-OWC', 'US-Ha1', 'US-KFS',
           'US-CF4', 'US-xBR', 'CA-TPD', 'US-CS1', 'CA-Cbo', 'US-Sne', 'MX-Tes',
           'US-Ro5', 'US-Bi1', 'US-ONA', 'US-Ro1', 'US-CF1', 'US-Ro4', 'CA-DBB',
           'US-Tw1', 'US-Rwf', 'US-Rms', 'US-NC1', 'US-CS3', 'US-Snf', 'US-GLE',
           'US-A32', 'CA-TP3', 'US-UM3', 'CA-ER1', 'US-Jo2', 'US-Hn2', 'US-CF3',
           'US-Jo1', 'US-Myb', 'US-HB1', 'US-MMS', 'US-NR1', 'US-Ro6', 'US-CS2',
           'US-Ne1', 'US-Tw5', 'US-ORv', 'US-UMd', 'US-KLS'],
          dtype='object', name='site'))

• Attributes: (0)

Let's save the dataset so we don't have to recompute it later!

In [5]:

ds.to_netcdf("/home/bart/Data/EXCITED/NEE_ameriflux_transcom2.nc")

ds.to_netcdf("/home/bart/Data/EXCITED/NEE_ameriflux_transcom2.nc")

And we can plot the NEE for a site (here grouped by DOY):

In [6]:

ds.isel(site=0)["NEE_VUT_REF"].groupby(ds.time.dt.dayofyear).mean().plot()

ds.isel(site=0)["NEE_VUT_REF"].groupby(ds.time.dt.dayofyear).mean().plot()

Out[6]:

[<matplotlib.lines.Line2D at 0x7f4bdf8b1d20>]

We can also easily compare the GPP to the NEE:

In [7]:

ds_site = ds.isel(site=0)
ds_site["GPP_NT_VUT_REF"].groupby(ds_site.time.dt.hour).mean().plot(label="GPP NT")
ds_site["GPP_DT_VUT_REF"].groupby(ds_site.time.dt.hour).mean().plot(label="GPP DT")
(-ds_site["NEE_VUT_REF"].groupby(ds_site.time.dt.hour).mean()).plot(label="NEE")
plt.legend()
plt.plot([0,23],[0,0],'k')

ds_site = ds.isel(site=0) ds_site["GPP_NT_VUT_REF"].groupby(ds_site.time.dt.hour).mean().plot(label="GPP NT") ds_site["GPP_DT_VUT_REF"].groupby(ds_site.time.dt.hour).mean().plot(label="GPP DT") (-ds_site["NEE_VUT_REF"].groupby(ds_site.time.dt.hour).mean()).plot(label="NEE") plt.legend() plt.plot([0,23],[0,0],'k')

Out[7]:

[<matplotlib.lines.Line2D at 0x7f4bdebe4760>]

Or look at the yearly pattern of respiration:

In [8]:

(ds_site["GPP_DT_VUT_REF"].groupby(ds_site.time.dt.dayofyear).mean()--ds_site["NEE_VUT_REF"].groupby(ds_site.time.dt.dayofyear).mean()).plot()

(ds_site["GPP_DT_VUT_REF"].groupby(ds_site.time.dt.dayofyear).mean()--ds_site["NEE_VUT_REF"].groupby(ds_site.time.dt.dayofyear).mean()).plot()

Out[8]:

[<matplotlib.lines.Line2D at 0x7f4bdeb2f640>]

In [ ]: