I like to divide such unfilmable films into 'couldn't' and 'shouldn't'.
For an example of 'couldn't', I would consider House of Leaves to be unfilmable in its entirety simply because of the nature of what's involved. It's literature that is virtually impossible to film.
And then there is 'shouldn't', novels that should never be adapted, not simply because they would be crappily adapted, but because they are crappy in the first place. Atlas Shrugged (despite it currently being made as an independent film trilogy) is a case in point.
Most books shouldn't be adapted into film. Watchmen was as close as a film adaptation could get to capturing the comic, but I read somewhere once that someone considered a miniseries, like A Game of Thrones.
Yami: Wanna play a game?
Ushio: What sort of game-
Yami: First one to die loses!
Yami: Look! the Queen of Spades!
Ushio: What does that do-
Yami: You lose!
Ushio: *falls* UAHHHHHHHHH!
-YuGiOh Season Zero Abridged