Multi tool use

Training Inception V3 based model using Keras with Tensorflow Backend

I am currently training a few custom models that require about 12Gb GPU memory at the most. My setup has about 96Gb of GPU memory and python/Jupyter still manages to hog up all the gpu memory to the point that I get the Resource exhausted error thrown at me. I am stuck at this peculiar issue for a while and hence any help will be appreciated.

Now, when loading a vgg based model similar to this:

from keras.applications.vgg16 import VGG16
from keras.models import Model

import keras

from keras.models import Model, Sequential
from keras.models import Input
input_shape = (512, 512, 3)
base_model = VGG16(input_shape=input_shape, weights=None, include_top=False)

pixel_branch = base_model.output
pixel_branch = Flatten()(pixel_branch)

new_model = Model(inputs=base_model.input, outputs=pixel_branch)

text_branch = Sequential()
text_branch.add(Dense(32, input_shape=(1,), activation='relu'))

# merged = Merge([new_model, text_branch], mode='concat')
merged = keras.layers.concatenate([new_model.output, text_branch.output])

age = Dense(1000, activation='relu')(merged)
age = Dense(1000, activation='relu')(age)
age = Dense(1)(age)

# show model
# model.summary()
model = Model(inputs=[base_model.input, text_branch.input], outputs=age)


When I just run a jupyter cell with this code and monitor the GPU usage using nvidia-smi, it is 0% .
However, I replace the code in the above Jupyter cell with the following:

from keras.applications.inception_v3 import InceptionV3
from keras.models import Model
import keras
from keras.models import Model
from keras.models import Sequential

from keras.models import Input
input_shape = (512, 512, 3)
base_model = InceptionV3(input_shape=input_shape, weights=None, include_top=False)

pixel_branch = base_model.output
pixel_branch = Flatten()(pixel_branch)

new_model = Model(inputs=base_model.input, outputs=pixel_branch)

text_branch = Sequential()
text_branch.add(Dense(32, input_shape=(1,), activation='relu'))

# merged = Merge([new_model, text_branch], mode='concat')
merged = keras.layers.concatenate([new_model.output, text_branch.output])

age = Dense(1000, activation='relu')(merged)
age = Dense(1000, activation='relu')(age)
age = Dense(1)(age)

# show model
# model.summary()
model = Model(inputs=[base_model.input, text_branch.input], outputs=age)


The GPU usage goes crazy and suddenly almost all the memory is over in all the GPUs even before I do model.compile() or model.fit() in Keras!

I have tried both allow_growth and per_process_gpu_memory_fraction in Tensorflow as well. I still get the resource exhausted error the moment I run model.fit when using the Inception based model.
Please note that I do not think this is a GPU memory error as I have about 96GB of GPU memory using an instance with 8 Tesla K80s .

Also kindly note that my batch size is 2.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.