Run the following code in your Python shell with Keras Python installed,

from keras.layers import Input, Embedding
from keras import backend as K
a = Input((12, ))
x = Embedding(1000, 10, trainable=False, input_length=10, name="word_embedding")(a)
print(K.int_shape(x))
print(x.get_shape().as_list())

And you’ll get suprised. You’ll get two different returns:

(None, 10, 10)
[None, 12, 10]

If you are not sure why, this article is for you!

Static Shapes

In Tensorflow, the static shape is given by .get_shape() method of the Tensor object, which is equivalent to .shape.

The static shape is an object of type tensorflow.python.framework.tensor_shape.TensorShape.

With None instead of an integer, it leaves the possibility for partially defined shapes :

a = Input((None, 10))
print(a.shape)

returns (?, ?, 10) where unknown dimensions None are printed with an question mark ?. Since we are using the Keras Input layer, the first dimension is systematically None for the batch size and cannot be set.

The TensorShape has 2 public attributes, dims and ndims:

print(a.shape.dims) # [Dimension(None), Dimension(None), Dimension(10)]
print(a.shape.ndims) # 3
print(K.ndim(a)) # 3

Dynamic Shapes

Of course, during the run of the graph with input values, all shapes become known. Shape at run time are named dynamic shapes.

To access to their values at run time, you can use either the tensorflow operator tf.shape() or the Keras wrapper K.shape().

As graph operators, they both return a tensorflow.python.framework.ops.Tensor Tensor.

a = Input((None, ))
import numpy as np
f = K.function([a], [K.shape(a)])
print(f( [np.random.random((3,11)) ]))

returns [array([ 3, 11], dtype=int32)]. 3 is the batch size, and 11 the value for the unknown data dimension at graph definition.

The equivalent of a.get_shape().as_list() for static shapes is tf.unstack(tf.shape(a)).

The number of dimensions is returned by tf.rank() which returns a tensor of rank zero (scalar) and for that reason is very different from K.ndim or ndims methods with integer return.

Shape setting for Operators

Most operators have an output shape function that enables to infer the static shape, without running the graph, given the shapes of the operators’ input tensors.

For example, in Tensorflow you can check how to define the Shape functions in C++ for any operator.

Nevertheless, shape functions do not cover all cases correctly, and in some cases, it is impossible to infer the shape without knowing more of your intent

Let’s take an example, where automatic shape inference is not possible.

Let’s define a reshaping operation given a reshaping tensor for which the values are not known at graph definition, but only at run time.

Such a reshaping tensor can be for example depending on input tensor dimensions, or any other dynamic shapes:

a = Input((None,))
reshape_tensor = Input((), dtype="int32")
print(a.shape)

# reshape the first input tensor given the second input tensor
x = K.reshape(a, reshape_tensor)
print(x.shape)

# build the model
f = K.function([a, reshape_tensor], [x])

# eval the model on input data
import numpy as np
f([np.random.random((3,10)), [3, 2,5] ])

prints

(?, ?)
<unknown>

The shape of variable a is of rank 2, but both dimensions are not known: it is named partially known shape. Its value is TensorShape([None, None]).

The shape of variable x is unknown, neither the rank nor the dimensions are known. Its value is TensorShape(None).

That is where, if possible, setting the shape manually can help get a more precise shape than TensorShape([None, None]) or TensorShape(None).

To set the unknown shape dimensions:

a = Input((None, 10))
a.set_shape((10,10,10))
print(a.shape)

Note that shape setting requires to preserve the shape rank and known dimensions. IF I write:

a.set_shape((10,10,10))

it leads to a value error

ValueError: Shapes (?, ?, 10) and (10, 10, 11) are not compatible

Setting the shape enables further operators to compute the shape.

Since Tensorflow does not have a concept of Layers (it is much more based on Operators and Scopes), the set_shape() function is the method for shape inference without running the graph.

Shape setting for Layers

Let’s come back to the initial example, where a layer, the Embedding layer, is a concept involved in the middle of the Keras graph definition.

The concept of layers gives a struture to the neural networks, enabling to run through the layers later on:

from keras.models import Model
a = Input((11,))
x = Embedding(1000, 10, trainable=False, input_length=10, name="word_embedding")(a)
m = Model(a,x)
for l in m.layers:
    print(l.name)

returns the name of each layers: input_1, word_embedding.

Still, when a shape cannot be inferred, it is possible to set it also, so that further layers benefit from their output shape information.

Let’s see in practice, with a simple custom concatenate Layer.

For the purpose, let me introduce an error in the compute_output_shape() function, adding 2 to the last shape dimension, as I did in the Embedding layer at the begining of this article, setting input_length to 10 instead of 12:

from keras.engine.topology import Layer
class MyConcatenateLayer(Layer):

    def call(self, inputs):
        return K.concatenate(inputs)

    def compute_output_shape(self, input_shape):
        print(input_shape) # [(None, 10), (None, 12)]
        return (None, input_shape[0][1] + input_shape[1][1] + 2)

from keras.layers import Input, Embedding
from keras import backend as K
a = Input((10,))
b = Input((12,))
c = MyConcatenateLayer()([a,b])
print(c.shape) # (?, 22)
print(c._keras_shape) # (?, 24)

The code will run without error, as well as the graph evaluation on values for inputs.

As you can see, Keras adds attributes to Tensor such as _keras_shape, to be able to retrieve layers information. This can be useful for layer weight saving for example.

The Keras K.int_shape() method relies on _keras_shape attribute to return the result, leading to propagation of the error.

Since shapes can vary in rank and their values can be None, it is difficult, on such a simple concatenation example, to be sure to cover all cases in the shape inference function and this leads to errors.

A note on CNTK

CNTK distinguishes unknown dimensions into 2 categories:

  • the inferred dimensions whose value is to be inferred by the system and is printed with -1 instead of the question mark ?. For example, in the matrix multiplication A x B between tensors A and B, the last dimension of A can be inferred by the system given the first dimension of B. See here

  • the free dimensions whose value is known only when data is bound to the variable and is printed with -3

Well done!

Now, you are aware of the Why of the shape setting, its advantages and its risks.