#server 2016

While we're still doing a postmortem and traipsing through event logs from the server between other problems as time allows, it's worth sharing that this appears to be a known issue and PC has a KB entry for it here: Managing the amount of memory used by PaperCut NG/MF | PaperCut

Sharing it below as well for posterity:

Managing the amount of memory used by PaperCut NG/MF

Note: This article is also useful if you’re wanting to limit the amount of memory that PaperCut uses - for example if you’re running PaperCut on a physical host with tens of GB of memory. Check out the ‘You may also specify the precise amount of physical RAM to use’ section below.

By default, the PaperCut NG and PaperCut MF Application Server is restricted to a maximum RAM allocation of up to one quarter (¼) of system memory. This ensures that the application does not consume too many resources, and “plays nicely” with other applications that may be run on the same system.

If the server is dedicated to running our software, then you may choose to increase the amount of memory made available to the Application Server component of the solution. This can improve application performance, particularly on sites with many users or with large printing volumes. Increased RAM allocation can help the solution by allowing it to perform optimizations, such as disk and data caching. Being database driven applications, PaperCut NG and PaperCut MF can benefit greatly from such enhancement.

A popular option with our customers is to allow the application to use up to ½ of the machine’s memory, and then provision the system with at least 4 GB of total memory. This can strike a good balance between giving the application enough resources to process and analyze jobs, as well as letting the machine handle the operating system level tasks required for a print server. Since server loads and performance vary greatly, we highly recommend testing the configuration and increasing allocated memory (and CPU cores) if you are seeing performance issues that you suspect are due to low resources.

Please Note: Somewhat counterintuitively, more is not always better; allocating the host of the Application Server an excessive amount of total system memory can in some cases have a deleterious effect on performance. PaperCut NG and PaperCut MF are Java applications, and hence subject to Java “garbage collection” activities, which maintain proper memory allocation. When a great deal of memory is available to the Application Server, full garbage collections will happen less frequently, but take longer to complete. The application must pause temporarily whilst these operations complete, so if the amount of memory available is imbalanced by a comparatively small amount of CPU resources, this may be observed as periodic spikes in highly unresponsive behavior. If unsure of the appropriateness of your resourcing, our Server Sizing Guide can be invaluable to check with. Increasing Maximum Memory (Windows/Linux/iPrint) To increase the amount of memory available to PaperCut NG or PaperCut MF: 1. Locate the text file called service.conf in the following directory:[app-path]\server\custom\ Please Note: If you’re running a version of PaperCut NG or PaperCut MF prior to 17.3, you will instead need to create the service.conf file in the directory noted above. 2. Open the file in a text editor and add the following line to allow the application to employ up to one third of total physical RAM: wrapper.java.additional.6=-XX:MaxRAMFraction=3 To employ up to one half of total physical RAM, use the line: wrapper.java.additional.6=-XX:MaxRAMFraction=2 You may also specify the precise amount of physical RAM to use. For example, to allow use of up to 768 MB, the following line can be added: wrapper.java.additional.6=-Xmx768m 3. Save and close the text file. 4. Restart the PaperCut Application Server service. On Windows, this can be done via Control Panel->Administration Tools->Services by locating the PaperCut Application Server in the list of system services, right-clicking, and selecting to Restart it. 5. After the service is restarted, the Application Server log file ([app-path]\server\logs\server.log) will show the new “max memory” setting. Increasing Maximum Memory (macOS) Please Note: This method only works on version 9.6 and higher of PaperCut NG and PaperCut MF. To increase amount of memory available to PaperCut NG or PaperCut MF on macOS: 1. Locate the text file called launch-app-server.conf in the following directory:[app-path]\server\custom\ Please Note: If you’re running a version of PaperCut NG or PaperCut MF prior to 17.3, you will instead need to create the launch-app-server.conf file in the directory noted about. Be sure to change the file permissions to make the file readable to all users, or at minimum, the papercut host user account. 2. You will need to specify the amount of system memory available to the Application Server. Open the file in a text editor and add the following line to allow the application to employ up to, for example, 256 MB of physical RAM: PC_CUSTOM_SERVER_ARG=-Xmx256m For up to 512 MB of physical RAM, you would instead use the line: PC_CUSTOM_SERVER_ARG=-Xmx512m 3. Save and close the text file. 4. Restart the PaperCut Application Server service by running the stop-server.command script and then the start-server.command script, both located in the following directory:[app-path]/server/bin/mac/ 5. After the service is restarted, the Application Server log file ([app-path]\server\logs\server.log) will show the new “max memory” setting. Running out of Memory or High CPU Usage If PaperCut NG or PaperCut MF run out of memory, an OutOfMemoryError will be displayed to the user and/or logged to the service.log file. You may also see GC overhead limit exceeded error, which is again an indication of running out of memory. Another indication of running out of memory is very high CPU usage for extended periods of time, by the pc-app process/service, as the service tries to rearrange RAM for various tasks. If this occurs then this is an indication that the server requires more memory, and

there are two possible solutions: Allocate more RAM to the server. Increase the maximum amount of memory available to the Application Server, as described above. This should typically only be done if there are no other applications deployed to the same server, or if recommended by the Support Team.

Now, having shared that, the issue also appeared related to how the vNIC was being handled by the VM servers situated on the host. Possibly two extremely different issues, but both still causing problems first thing on a Monday morning. The issue specifically related to the VM print server being inaccessible via RDP and being unable to be pinged. It could, however, be accessed via SCVMM and Hyper-V manager through the console. Strange, right?

Rebooting the server failed to immediately address the problem, so we set to triaging after sending out an announcement to the affected user base. After discussing as a team, turns out this had been seen elsewhere recently and it was indeed a vNIC issue. What resolved the clear and present trouble was to shut down the VM and go into Hyper-V Manager >> right-click on the given server >> Settings...

>>  Under Settings >> Network Adapter Options >> Change it from Global Converged Fabric Switch to Not Connected >> Click Apply and OK

then shut the VM down (it won't matter since the VM is on the fritz anyway, right?) Great.

Once the VM has been shut down, change it back to Global Converged Fabric Switch and power the VM back up in Hyper-V Manager.

This should resolve the issue and can be checked and confirmed by running ping SERVERNAME -t in CMD or using test-connection -ComputerName SERVERNAME, connecting to the VM via RDP, or attempting to access the offending resource once the VM is back up and you've been able to log into it using your Windows account.