POST with large content fails

[wbrooks Profile]

I am having trouble with git posting large change sets to scm-manager.

The server is using the following components:
Windows 7
IIS 7.5
Helicon Zoo 3.1.98.530
Jetty 7
scm-manager 1.36

When attempting to push a large git repository, git eventually fails with:

User-Agent: git/1.9.2.msysgit.0
Host: *********
Accept-Encoding: gzip
Content-Type: application/x-git-receive-pack-request
Accept: application/x-git-receive-pack-result
Content-Length: 44588941

* SSL read: error:00000000:lib(0):func(0):reason(0), errno 10054
* Closing connection 1
error: RPC failed; result=56, HTTP code = 0
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

As you can see from the Content-Length header, the POST entity is about 44MB in size. My initial assumption was the Jetty was likely refusing the request because of the content size. I have attempted to change the Jetty configuration to allow unlimited post content. In order to adjust the POST content size, I added the following to the etc\jetty.xml configuration file within the Jetty7 installation directory.

<Call name="setAttribute">
<Arg>org.eclipse.jetty.server.Request.maxFormContentSize</Arg>
<Arg>-1</Arg>
</Call>

This did not change the results from those listed above. I do not know for sure that the content size change has taken effect, as I do not know how to verify within Jetty that the change was accepted. I also verified that the IIS configuration maximum POST size is sufficiently large to accept the request.

I have verified that the POST request appears in both the IIS and Jetty request logs. The Jetty stderr log contains the following errors:

2014-06-11 15:25:03.629:WARN:/scm:Internal error during receive-pack
org.eclipse.jetty.io.EofException: early EOF
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65)
at org.eclipse.jgit.transport.PackParser.fill(PackParser.java:1129)
at org.eclipse.jgit.transport.PackParser.access$000(PackParser.java:97)
at org.eclipse.jgit.transport.PackParser$InflaterStream.read(PackParser.java:1661)
at java.io.InputStream.read(InputStream.java:101)
...

This exception leads me to believe that the input socket has been closed by Helicon Zoo. Continuing with this theory, I enabled Failed Request Tracing within IIS and found the following notifications:

3322. +GENERAL_READ_ENTITY_START
3323. -GENERAL_READ_ENTITY_END 142601 ms Informational
BytesReceived 0
ErrorCode 2147943395
ErrorCode The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3)

3324. -NOTIFY_MODULE_COMPLETION Verbose
ModuleName HeliconZoo_x64
Notification 128
fIsPostNotificationEvent false
CompletionBytes 0
ErrorCode 2147943395
Notification EXECUTE_REQUEST_HANDLER
ErrorCode The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3)

3325. -NOTIFY_MODULE_END Verbose 142601 ms
ModuleName HeliconZoo_x64
Notification 128
fIsPostNotificationEvent false
NotificationStatus 0
Notification EXECUTE_REQUEST_HANDLER
NotificationStatus NOTIFICATION_CONTINUE

I continued further with a packet capture on the client side. The client sends packets as expected, ultimately receiving a TCP frame with the RESET flag set.

All of this together suggests to me that either IIS or Helicon Zoo is terminating the request for some reason. I do not know how to continue to prove or disprove this theory and would appreciate any direction or solution.

[Anton]

Hello,

The problem might be due to IIS limitation.
Please go to IIS -> target website -> Request Filtering -> Edit Feature Settings (in right sidebar) and you will see the Maximum Allowed Content Length parameter in bytes. Please make sure the value is large enough and increase it if necessary.

[wbrooks Profile]

Thank you for the suggestion.

I had previously checked this setting. The value is (and was) 3221225472.

[Anton]

Hello,

Please create a folder in site dir for cache and add the following lines in web.config:

<add name="POST_CACHE_DIR" value="<cache_folder>" />
<add name="POST_BUFFER" value="0" />

If it doesn't work, please try changing the last line to

<add name="POST_BUFFER" value="1024" />

[wbrooks Profile]

Thank for the suggestion.

I added the following to the web.config file:

<add name="POST_CACHE_DIR" value="%APPL_PHYSICAL_PATH%/.cache" />
<add name="POST_BUFFER" value="0" />

The git push continues to fail with the same error indicated in my original post.

Changing the web.config to include:

<add name="POST_BUFFER" value="1024" />

Results in a 403 error.

[Ruslan]

Hello,

Thank you for reporting.
We have reproduce and confirm the issue and will investigate it.

We will inform you about results.

[Anton]

Hello,

We've fixed the issue.
Please update your Zoo from beta-feed now: http://www.helicontech.com/zoo/feed-beta/

[wbrooks Profile]

Hello,

I've updated Helicon Zoo to version 3.1.98.534. Unfortunately, the problem persists.

I have retested with and without the POST_CACHE_DIR and POST_BUFFER values.

[Anton]

Hello,

This is strange.
Please make sure the dll has been updated and IIS has been restarted.
If it's really the latest build, please add this line into web.config

<add name="ERROR_LOG_LEVEL" value="trace" />

Make one test git push (to reproduce the situation) and provide us with the logs you will get.

[wbrooks Profile]

I have added the "ERROR_LOG_LEVEL" value into the web.config, but I am having difficultly locating the log files.

What files should I be looking for?

[Anton]

Hello,

ERROR_LOG_DIR variable contains the path to the log file. By default it's site root.
And make sure the user account has Write permissions to write into the file.