This initially sounds similar to other reports of Pascal-family GPU issues, for example:

https://bugs.freedesktop.org/show_bug.cgi?id=100228
https://bugs.freedesktop.org/show_bug.cgi?id=101665
https://bugs.freedesktop.org/show_bug.cgi?id=101220
https://bugs.freedesktop.org/show_bug.cgi?id=104421


What version of the kernel are you using? Upstream v4.14 (and soon to be released v4.15) have some improvements that should benefit you. Can you also provide the exact error text that you see.

You do mention that "setting nomodeset" doesn't improve things. This is interesting as it does indicate you may be seeing a different issue here.

Can you try with "nouveau.runpm=0" as a diagnostic step. This may let you boot, although power usage would be increased.

Hi again, i can boot with nouveau.runpm=0 although Linux Mint 18.2 didn't require an adjustment like that.

Sys info;

inxi -SMGxx
System:    Host: mint Kernel: 4.10.0-38-generic x86_64 (64 bit gcc: 5.4.0)
           Desktop: Cinnamon 3.6.6 (Gtk 3.18.9-1ubuntu3.3) dm: lightdm
           Distro: Linux Mint 18.3 Sylvia
Machine:   System: ASUSTeK (portable) product: GL553VD v: 1.0
           Mobo: ASUSTeK model: GL553VD v: 1.0
           Bios: American Megatrends v: GL553VD.304 date: 09/07/2017
Graphics:  Card-1: Intel Device 591b bus-ID: 00:02.0 chip-ID: 8086:591b
           Card-2: NVIDIA Device 1c8d bus-ID: 01:00.0 chip-ID: 10de:1c8d
           Display Server: X.Org 1.18.4 drivers: intel (unloaded: fbdev,vesa) FAILED: nouveau
           Resolution: 1920x1080@60.01hz
           GLX Renderer: Mesa DRI Intel Kabylake GT2
           GLX Version: 3.0 Mesa 17.0.7 Direct Rendering: Yes

inxi -Fxz
System:    Host: mint Kernel: 4.10.0-38-generic x86_64 (64 bit gcc: 5.4.0)
           Desktop: Cinnamon 3.6.6 (Gtk 3.18.9-1ubuntu3.3) Distro: Linux Mint 18.3 Sylvia
Machine:   System: ASUSTeK (portable) product: GL553VD v: 1.0
           Mobo: ASUSTeK model: GL553VD v: 1.0 Bios: American Megatrends v: GL553VD.304 date: 09/07/2017
CPU:       Quad core Intel Core i7-7700HQ (-HT-MCP-) cache: 6144 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 22464
           clock speeds: max: 3800 MHz 1: 899 MHz 2: 899 MHz 3: 1911 MHz 4: 899 MHz 5: 899 MHz 6: 1450 MHz
           7: 899 MHz 8: 1250 MHz
Graphics:  Card-1: Intel Device 591b bus-ID: 00:02.0
           Card-2: NVIDIA Device 1c8d bus-ID: 01:00.0
           Display Server: X.Org 1.18.4 drivers: intel (unloaded: fbdev,vesa) FAILED: nouveau
           Resolution: 1920x1080@60.01hz
           GLX Renderer: Mesa DRI Intel Kabylake GT2 GLX Version: 3.0 Mesa 17.0.7 Direct Rendering: Yes
Audio:     Card Intel Device a171 driver: snd_hda_intel bus-ID: 00:1f.3 Sound: ALSA v: k4.10.0-38-generic
Network:   Card-1: Intel Wireless 7265 driver: iwlwifi bus-ID: 02:00.0
           IF: wlp2s0 state: down mac: <filter>
           Card-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           driver: r8169 v: 2.3LK-NAPI port: d000 bus-ID: 03:00.0
           IF: enp3s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
Drives:    HDD Total Size: NA (-) ID-1: /dev/sda model: SanDisk_SD8SN8U1 size: 128.0GB temp: 40C
           ID-2: /dev/sdb model: HGST_HTS541010A9 size: 1000.2GB temp: 30C
           ID-3: USB /dev/sdc model: silicon size: 4.0GB temp: 0C
RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors:   System Temperatures: cpu: 51.0C mobo: N/A
           Fan Speeds (in rpm): cpu: 0
Info:      Processes: 249 Uptime: 17 min Memory: 876.9/7866.2MB Init: systemd runlevel: 5 Gcc sys: 5.4.0
           Client: Shell (bash 4.3.481) inxi: 2.2.35 

So my guess is something has changed during this time period ; either firmwares or Mesa version and caused this.

X was the same with no problem booting 18.2 and with 18.3.

So problem might not be X related at all. 

I will add this drm pointer error also.

Created attachment 136736 [details]
Error - Drm

That is the error.

I gave the sys , used kernel etc. infos already.

Based on the kernel backtrace and system this is a duplicate of a bug report already (bz#100228) that tracks support for the GP107/nv137 chip. Take a look there for the software packages that you will need and ongoing development work.

Given you can run with 'nouveau.runpm=0', you likely have all of the current required software packages for *preliminary* support for GP10x. The prior downstream kernel shipped by Linux Mint 18.2 probably lacked any software support attempting the bring up the GP107 graphics card.

As outlined by Ilia Mirkin, you will need at minimum (numbers are the mainline version or Debian packages, not sure which kernel fixes/features Arch backports):

Linux kernel >= 4.12
xf86-video-nouveau >= 1.0.15
linux-firmware >= 1.165
mesa git

Double check your linux-firmware as there was incorrect Pascal firmware loaded during early 2017.

*** This bug has been marked as a duplicate of bug 100228 ***