Definitions of hard time:

• noun:   a difficulty that can be overcome with effort

  Example: "We had a hard time getting here"

• noun:   a term served in a maximum security prison