I don’t think this particular use case would work. TRAVIS_PULL_REQUEST is defined on the build worker, while build matrix must be constructed before Travis hands off the job to the worker.
I suggest writing a wrapper, so basically a script that takes TRAVIS_PULL_REQUEST and sets the environment variables correctly, or do something like this in before_install:
[ "${TRAVIS_PULL_REQUEST}" != "false" ] && BROWSER='chrome_linux' BUILD='default' || true