Package upload problems

[overcasi]

Whenever we upload a large package (~10GB+) via the web interface (we've stopped using casper admin) that is above a certain size, we have issues in the replication process between the JSS and the JDS.

The upload looks to have completed successfully, but upon looking in the JDS share directory, it never arrives.

The mysql database is holding the package in the downloadable_file_chunk_data.MYD table, which has grown to 30GB.

The problem seems to be that when the JDS checks in to replicate the package to the JDS share directory, it encounters an error in pulling it down:

2014-08-07 10:52:07,970 INFO Collecting inventory...
2014-08-07 10:52:08,382 INFO Sending inventory
2014-08-07 10:52:08,888 INFO Finished policies
2014-08-07 10:55:01,729 INFO Checking for policies...
2014-08-07 10:55:03,206 INFO Received 3 responses.
2014-08-07 10:55:03,206 INFO Adding package "Base-10-9-4.dmg" for download
2014-08-07 10:55:03,207 INFO Adding package "tmp-testingonly-nukeme.dmg" for download
2014-08-07 10:55:03,217 INFO Downloading package "Base-10-9-4.dmg"
2014-08-07 10:56:02,771 ERROR Failed to download package "Base-10-9-4.dmg"
2014-08-07 10:56:02,772 ERROR (23, 'Failed writing body (2073 != 8999)')
Traceback (most recent call last):
  File "<string>", line 781, in download
error: (23, 'Failed writing body (2073 != 8999)')

the servers are both running on RHEL 6.5

[overcasi]

This is the error in the JAMFSoftwareServer.log, the previous post's error is from jamf.log on the JDS.

2014-08-07 13:57:05,765 [ERROR] [Tomcat-346 ] [ibutionServerFileDownload] - Error reading file with ID 8 - com.jamfsoftware.jss.exceptions.downloadablefiles.FileWriteException: ClientAbortException:  java.net.SocketException: Connection reset
com.jamfsoftware.jss.exceptions.downloadablefiles.FileWriteException: ClientAbortException:  java.net.SocketException: Connection reset
        at com.jamfsoftware.jss.objects.downloadablefile.StreamConsumer.consume(StreamConsumer.java:35)
        at com.jamfsoftware.jss.objects.downloadablefile.DownloadableFile.load(DownloadableFile.java:155)
        at com.jamfsoftware.jss.objects.downloadablefile.DownloadableFile$ThreadedReader.run(DownloadableFile.java:217)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: ClientAbortException:  java.net.SocketException: Connection reset
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:371)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:438)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:426)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:84)
        at com.jamfsoftware.jss.objects.downloadablefile.StreamConsumer.writeToStream(StreamConsumer.java:41)
        at com.jamfsoftware.jss.objects.downloadablefile.StreamConsumer.consume(StreamConsumer.java:33)
        ... 7 more
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
        at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
        at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
        at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
        at org.apache.coyote.Response.doWrite(Response.java:520)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)
        ... 14 more

[lynnp]

For what it's worth, we solved this problem by eliminating the seperate partition for /tmp on our JDS. It only had 2GB allocated (which was a standard setup for our RHEL install) and the JDS actually writes to this location when replicating.

It's worth noting that the path a package takes when uploaded is as follows: Local storage -> Tomcat temp dir -> mysql db on the JSS -> JDS /tmp -> JDS storage dir

[lynnp]

Why the package data needs to touch the mysql db is still a mystery to me.