apache-hamilton-1.89.0 Released

[Stefan Krawczyk <st...@gmail.com>]

Hi All,

Our first Apache Hamilton release just dropped -- 1.89.0.

   - Source to directly download from
   <https://dist.apache.org/repos/dist/release/incubator/hamilton/1.89.0/>
   (we'll have official pypi apache-hamilton target shortly).
   - Release Notes
   <https://github.com/apache/hamilton/releases/tag/v1.89.0-incubating>
   - Note: pip install sf-hamilton==1.89.0 will continue to work.

We'll be releasing all the other Apache Hamilton packages over the next few
months as we tick all the boxes needed to release them officially under
Apache.

# What's new?

Docs -- hamilton.apache.org with Apache links.

Features:

   - Updated the existing function modifier @extract_fields so that it can
   infer field types from the type annotation.
   - Added a new modifier called @unpack_fields which allows for the
   extraction of fields from a tuple output (a cross between extract_columns
   and extract_fields).

Fixes:

   - dlt loader -- dlt changed their library, this fixes that


*##  Updated the existing function modifier extract_fields so that it can
infer field types from the type annotation.*
To use this feature you must specify a generic dictionary with valid type
paramerters - therefore it will only work for homogenous dictionaries. For
example, the following would extract the standard X_train, X_test, y_train,
and y_test as np.ndarray by using unpacked field names:

@extract_fields('X_train', 'X_test' 'y_train' 'y_test')  # unpacked field
names
def train_test_split_func(...) -> Dict[str, np.ndarray]:
    ...
    return {"X_train": ..., "X_test": ..., "y_train": ..., "y_test": ...}

You can also pass a list of field names to the first argument:

@extract_fields(['X_train', 'X_test' 'y_train' 'y_test'])   # list of field
names
def train_test_split_func(...) -> Dict[str, np.ndarray]:
    ...
    return {"X_train": ..., "X_test": ..., "y_train": ..., "y_test": ...}

This also preserves backward compatibility with non-generic dictionaries:

@extract_fields(dict(  # fields specified as a dictionary
    X_train=np.ndarray,
    X_validation=np.ndarray,
    X_test=np.ndarray,
))
def train_test_split_func(...) -> Dict:
    ...
    return {"X_train": ..., "X_test": ..., "y_train": ..., "y_test": ...}

*##  @unpack_fields*

Example usage with an fixed size tuple:

@unpack_fields("X_train" "X_validation", "X_test")
  def dataset_splits(X: np.ndarray) -> Tuple[np.ndarray, np.ndarray,
np.ndarray]:
      """Randomly split data into train, validation, test"""
      X_train, X_validation, X_test = random_split(X)
      return X_train, X_validation, X_test

Example usage with a subset of fixed length tuples:

@unpack_fields("X_train")
  def dataset_splits(X: np.ndarray) -> Tuple[np.ndarray, np.ndarray,
np.ndarray]:
      """Randomly split data into train, validation, test"""
      X_train, X_validation, X_test = random_split(X)
      return X_train, X_validation, X_test

Example usage with an indeterminate length tuple:

 @unpack_fields("X_train" "X_validation", "X_test")
  def dataset_splits(X: np.ndarray) -> Tuple[np.ndarray, ...]:
      """Randomly split data into train, validation, test"""
      X_train, X_validation, X_test = random_split(X)
      return X_train, X_validation, X_test

Thanks to Charles Swartz for the new features and many more folks around
helping get all the Apache requirements fulfilled for this first release
(thanks apache mentors!)

Cheers,

Stefan