Hijax

jaxhijaxdocs_nnxflax

Hijax

[1]
[2]
0.85250294
0.8165137
0.7814907

Hijax Variable

State propagation:

[3]
0
1
[4]
{ lambda ; a:Variable(). let
    jit[
      name=inc
      jaxpr={ lambda ; a:Variable(). let
          b:i32[] = get_variable[
            avals=(ShapedArray(int32[], weak_type=True),)
            has_qdd=True
            treedef=PyTreeDef(CustomNode(Variable[(('eager_sharding', True), ('hijax', True), ('mutable', True), ('ref', False))], [*]))
            var_type=<class 'flax.nnx.variablelib.Variable'>
          ] a
          c:i32[] = add b 1:i32[]
          _:i32[] = get_variable[
            avals=(ShapedArray(int32[], weak_type=True),)
            has_qdd=True
            treedef=PyTreeDef(CustomNode(Variable[(('eager_sharding', True), ('hijax', True), ('mutable', True), ('ref', False))], [*]))
            var_type=<class 'flax.nnx.variablelib.Variable'>
          ] a
          set_variable[
            treedef=PyTreeDef(CustomNode(Variable[(('eager_sharding', True), ('hijax', True), ('mutable', True), ('ref', False))], [*]))
            var_type=<class 'flax.nnx.variablelib.Variable'>
          ] a c
        in () }
    ] a
  in () }

Pytree values:

[5]
Variable( # 2 (8 B)
  value={'a': Array(0, dtype=int32, weak_type=True), 'b': Array(2, dtype=int32, weak_type=True)},
  hijax=True
)
Variable( # 2 (8 B)
  value={'a': Array(1, dtype=int32, weak_type=True), 'b': Array(4, dtype=int32, weak_type=True)},
  hijax=True
)

Dynamic state structure:

[6]
Before: Variable(
  value={},
  hijax=True
)
After: Variable( # 1 (4 B)
  value={'y_mean': Array(-1.1782329, dtype=float32)},
  hijax=True
)
[7]
Variable( # 3 (12 B)
  value=Array([1, 2, 3], dtype=int32),
  hijax=True
)

Mutability

[8]
nnx.vars_as(model, mutable=False) = Linear( # Param: 3 (12 B)
  kernel=Param( # 3 (12 B)
    value=Array(shape=(1, 3), dtype=dtype('float32')),
    hijax=True,
    mutable=False
  )
)
nnx.vars_as(model, mutable=True) = Linear( # Param: 3 (12 B)
  kernel=Param( # 3 (12 B)
    value=Array(shape=(1, 3), dtype=dtype('float32')),
    hijax=True
  )
)
[9]
ImmutableVariableError: Cannot mutate Variable as it is marked as immutable. (https://flax.readthedocs.io/en/latest/api_reference/flax.errors.html#flax.errors.ImmutableVariableError)

Ref support

[10]
Variable( # 1 (4 B)
  value=Array(0, dtype=int32, weak_type=True),
  hijax=True,
  ref=True
)
Ref(0, dtype=int32, weak_type=True)
[11]
immutable = Variable( # 1 (4 B)
  value=Array(0, dtype=int32, weak_type=True),
  had_ref=True,
  hijax=True,
  mutable=False
)
mutable = Variable( # 1 (4 B)
  value=Array(0, dtype=int32, weak_type=True),
  hijax=True,
  ref=True
)

Examples

[12]

Training Loop

[13]
loss = 1.000178
loss = 0.9700456
loss = 0.93967044

Scan Over Layers

[14]

Limitations

Mutable Outputs

[15]
Error: mutable hitypes should use lo_ty_qdd instead
[16]
model.linear = Linear( # Param: 128 (512 B)
  kernel=Param( # 128 (512 B)
    value=Array(shape=(2, 64), dtype=dtype('float32')),
    hijax=True
  )
)

Reference Sharing (aliasing)

[17]
None
[18]
None
HasShared( # Variable: 1 (4 B)
  a=Variable( # 1 (4 B)
    value=Array(0, dtype=int32, weak_type=True),
    hijax=True
  ),
  b=Variable( # 1 (4 B)
    value=Array(0, dtype=int32, weak_type=True),
    hijax=True
  )
)
[19]
Duplicates found:
- [('a',), ('b',)]
[20]
HasShared( # Variable: 1 (4 B)
  a=Variable( # 1 (4 B)
    value=Array(5, dtype=int32, weak_type=True),
    hijax=True
  ),
  b=Variable( # 1 (4 B)
    value=Array(5, dtype=int32, weak_type=True),
    hijax=True
  )
)
[21]