FastAI 40 Tabular.Core

40 Tabular.Core

colabmachine-learninggpudeep-learningnotebooksPythonnbsfastaipytorch

alph-notebooks/fastai / 40_tabular.core.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

[ ]

Tabular core

Basic function to preprocess tabular data before assembling it in a DataLoaders.

Initial preprocessing

[ ]

For example if we have a series of dates we can then generate features such as Year, Month, Day, Dayofweek, Is_month_start, etc as shown below:

[ ]

This function works by determining if a column is continuous or categorical based on the cardinality of its values. If it is above the max_card parameter (or a float datatype) then it will be added to the cont_names else cat_names. An example is below:

[ ]

cont_names: ['cont1', 'f16']
cat_names: ['cat1', 'cat2', 'i8', 'u8', 'y1', 'y2']`

[ ]

cont_names: ['ui32', 'i64', 'f16', 'd1_Year', 'd1_Month', 'd1_Week', 'd1_Day', 'd1_Dayofweek', 'd1_Dayofyear', 'd1_Elapsed']
cat_names: ['cat1', 'd1_date', 'd1_Is_month_end', 'd1_Is_month_start', 'd1_Is_quarter_end', 'd1_Is_quarter_start', 'd1_Is_year_end', 'd1_Is_year_start']

[ ]

For example we will make a sample DataFrame with int, float, bool, and object datatypes:

[ ]

i         int64
,f       float64
,e          bool
,date        str
,dtype: object

We can then call df_shrink_dtypes to find the smallest possible datatype that can support the data:

[ ]

{'i': dtype('int8'), 'f': dtype('float32'), 'date': 'category'}

[ ]

df_shrink(df) attempts to make a DataFrame uses less memory, by fit numeric columns into smallest datatypes. In addition:

boolean, category, datetime64[ns] dtype columns are ignored.
'object' type columns are categorified, which can save a lot of memory in large dataset. It can be turned off by obj2cat=False.
int2uint=True, to fit int types to uint types, if all data in the column is >= 0.
columns can be excluded by name using excl_cols=['col1','col2'].

To get only new column data types without actually casting a DataFrame, use df_shrink_dtypes() with all the same parameters for df_shrink().

[ ]

Let's compare the two:

[ ]

i         int64
,f       float64
,u         int64
,date        str
,dtype: object

[ ]

i          int8
,f       float32
,u         int16
,date        str
,dtype: object

We can see that the datatypes changed, and even further we can look at their relative memory usages:

[ ]

Initial Dataframe: 228 bytes
Reduced Dataframe: 177 bytes

[ ]

Here's another example using the ADULT_SAMPLE dataset:

[ ]

Initial Dataframe: 3.907452 megabytes
Reduced Dataframe: 0.814989 megabytes

We reduced the overall memory used by 79%!

Tabular -

[ ]

df: A DataFrame of your data
cat_names: Your categorical x variables
cont_names: Your continuous x variables
y_names: Your dependent y variables
- Note: Mixed y's such as Regression and Classification is not currently supported, however multiple regression or classification outputs is
y_block: How to sub-categorize the type of y_names (CategoryBlock or RegressionBlock)
splits: How to split your data
do_setup: A parameter for if Tabular will run the data through the procs upon initialization
device: cuda or cpu
inplace: If True, Tabular will not keep a separate copy of your original DataFrame in memory. You should ensure pd.options.mode.chained_assignment is None before setting this
reduce_memory: fastai will attempt to reduce the overall memory usage by the inputted DataFrame with df_shrink

[ ]

/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipymini_23754/1389180172.py:6: FutureWarning: `torch.distributed.reduce_op` is deprecated, please use `torch.distributed.ReduceOp` instead
  return len([x for x in objs if isinstance(x, pd.DataFrame)])

[ ]

These transforms are applied as soon as the data is available rather than as data is called from the DataLoader

[ ]

While visually in the DataFrame you will not see a change, the classes are stored in to.procs.categorify as we can see below on a dummy DataFrame:

[ ]

Each column's unique values are stored in a dictionary of column:[values]:

[ ]

{'a': ['#na#', np.int8(0), np.int8(1), np.int8(2)]}

[ ]

/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipymini_23754/3600165379.py:3: Pandas4Warning: Constructing a Categorical with a dtype and values containing non-null entries not in that dtype's categories is deprecated and will raise in a future version.
  return pd.Categorical(c, categories=voc[c.name][add:]).codes+add

[ ]

/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipymini_23754/3600165379.py:3: Pandas4Warning: Constructing a Categorical with a dtype and values containing non-null entries not in that dtype's categories is deprecated and will raise in a future version.
  return pd.Categorical(c, categories=voc[c.name][add:]).codes+add

[ ]

/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipymini_23754/3600165379.py:3: Pandas4Warning: Constructing a Categorical with a dtype and values containing non-null entries not in that dtype's categories is deprecated and will raise in a future version.
  return pd.Categorical(c, categories=voc[c.name][add:]).codes+add

[ ]

/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipymini_23754/3600165379.py:3: Pandas4Warning: Constructing a Categorical with a dtype and values containing non-null entries not in that dtype's categories is deprecated and will raise in a future version.
  return pd.Categorical(c, categories=voc[c.name][add:]).codes+add

[ ]

/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipymini_23754/3600165379.py:3: Pandas4Warning: Constructing a Categorical with a dtype and values containing non-null entries not in that dtype's categories is deprecated and will raise in a future version.
  return pd.Categorical(c, categories=voc[c.name][add:]).codes+add

[ ]

Currently, filling with the median, a constant, and the mode are supported.

[ ]

TabularPandas Pipelines -

[ ]

/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipymini_23754/3600165379.py:3: Pandas4Warning: Constructing a Categorical with a dtype and values containing non-null entries not in that dtype's categories is deprecated and will raise in a future version.
  return pd.Categorical(c, categories=voc[c.name][add:]).codes+add

[ ]

Integration example

For a more in-depth explanation, see the tabular tutorial

[ ]

/Users/jhoward/aai-ws/fastai/fastai/torch_core.py:154: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:212.)
  else as_tensor(x.values, **kwargs) if isinstance(x, (pd.Series, pd.DataFrame))

[ ]

We can decode any set of transformed data by calling to.decode_row with our raw data:

[ ]

age                                50.0
,workclass              Self-emp-not-inc
,fnlwgt                         124793.0
,education                       HS-grad
,education-num                       9.0
,marital-status       Married-civ-spouse
,occupation                 Craft-repair
,relationship                    Husband
,race                              White
,sex                                Male
,capital-gain                          0
,capital-loss                          0
,hours-per-week                       30
,native-country            United-States
,salary                             <50k
,education-num_na                  False
,Name: 3564, dtype: object

We can make new test datasets based on the training data with the to.new()

:::{.callout-note}

Since machine learning models can't magically understand categories it was never trained on, the data should reflect this. If there are different missing values in your test data you should address this before training

:::

[ ]

We can then convert it to a DataLoader:

[ ]

TabDataLoader's create_item method

[ ]

age    35
Name: 0, dtype: int8

Other target types

Multi-label categories

one-hot encoded label

[ ]

CPU times: user 30.8 ms, sys: 774 us, total: 31.6 ms
Wall time: 31.4 ms

[ ]

Not one-hot encoded

[ ]

CPU times: user 10.5 ms, sys: 201 us, total: 10.7 ms
Wall time: 10.6 ms

[ ]

['-', '_', 'a', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'o', 'p', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y']

Regression

[ ]

CPU times: user 21.8 ms, sys: 969 us, total: 22.7 ms
Wall time: 21.9 ms

[ ]

{'fnlwgt': np.float64(192511.077125),
, 'education-num': np.float64(10.076749801635742)}

[ ]

Not being used now - for multi-modal

[ ]

Export -

[ ]